ConvNeXt

ConvNeXt is a family of convolutional neural networks for image classification and dense prediction, introduced by Liu, Mao, Wu, Feichtenhofer, Darrell and Xie at CVPR 2022 in “A ConvNet for the 2020s”. It was not designed from scratch. The authors started from a ResNet-50 trained with a transformer-era recipe and changed it one design decision at a time toward the hierarchical Swin Transformer, adopting or rejecting each change in turn, until the network, built only from standard convolutional modules, matched or exceeded Swin at equal FLOPs. The result is a four-stage network with a patchify stem, $7 \times 7$ depthwise convolutions, inverted bottlenecks, a single GELU and a single LayerNorm per block, and separate downsampling layers. ConvNeXt V2 (Woo et al., CVPR 2023) kept this design, added a global response normalization layer and pre-trained it as a fully convolutional masked autoencoder.

Problem Addressed

By 2021 the Vision Transformer and its hierarchical successors had overtaken convolutional networks on ImageNet, and Swin Transformer had shown that a transformer with local windows and a multi-stage layout could serve as a generic backbone for detection and segmentation. Liu et al. argue that the comparison behind this shift was confounded. Comparisons of a Swin Transformer with a ResNet were system-level: the two differed in training procedure and in macro and micro architecture as well as in attention against convolution, and the performance difference was usually credited to the transformer’s superior scaling, with multi-head self-attention seen as the key component. Swin also brought back convolutional priors such as local windows and a feature hierarchy, at the cost of more complex modules such as cyclic shifting.

The paper asks how design decisions in transformers affect a ConvNet’s performance when everything else is held fixed. Its answer is the modernization roadmap: a sequence of networks, each trained with the same recipe on ImageNet-1K, that isolates one design choice per step.

Architecture

ConvNeXt keeps the four-stage hierarchy of ResNet and Swin. A stem reduces a $224 \times 224$ image to a $56 \times 56$ map; stages 1 to 4 run at strides 4, 8, 16 and 32 ($56^2$, $28^2$, $14^2$ and $7^2$ positions), the channel width doubles from stage to stage, and a global average pool and a linear classifier end the network. Each stage is a stack of identical residual blocks. The variants differ only in the stage widths $C$ and the number of blocks $B$ (§3):

Model	$C$	$B$	Params	FLOPs at $224^2$	ImageNet-1K top-1
ConvNeXt-T	(96, 192, 384, 768)	(3, 3, 9, 3)	29M	4.5G	82.1
ConvNeXt-S	(96, 192, 384, 768)	(3, 3, 27, 3)	50M	8.7G	83.1
ConvNeXt-B	(128, 256, 512, 1024)	(3, 3, 27, 3)	89M	15.4G	83.8
ConvNeXt-L	(192, 384, 768, 1536)	(3, 3, 27, 3)	198M	34.4G	84.3
ConvNeXt-XL	(256, 512, 1024, 2048)	(3, 3, 27, 3)	350M	60.9G	—

Parameters, FLOPs and accuracies are from Table 1 (ImageNet-1K training, $224^2$ input; ConvNeXt-XL was trained only with ImageNet-22K pre-training). ConvNeXt-T and ConvNeXt-B are the end points of the roadmap in the ResNet-50 and ResNet-200 regimes; the supplementary Table 9 gives ConvNeXt-T as $28.6 \times 10^6$ parameters and $4.5 \times 10^9$ FLOPs, against $25.6 \times 10^6$ and $4.1 \times 10^9$ for ResNet-50.

The Modernization Roadmap

The roadmap begins with a recipe change, not an architectural one. Trained for 300 epochs instead of 90, with AdamW, Mixup, CutMix, RandAugment, Random Erasing, stochastic depth and label smoothing, a recipe close to DeiT’s and Swin’s, ResNet-50 rose from 76.1% to 78.8% top-1 accuracy on ImageNet-1K. The authors infer that a significant portion of the earlier gap between ConvNets and vision transformers may be due to training techniques. Every later step uses this recipe unchanged, and each accuracy in the ResNet-50 regime is the mean of three runs with different seeds. The supplementary Table 10 lists every step with its accuracy (mean ± standard deviation) and GFLOPs:

Step	What changes	Top-1 (%)	GFLOPs
ResNet-50, improved recipe	baseline	78.82 ± 0.07	4.09
stage ratio	blocks per stage (3, 4, 6, 3) → (3, 3, 9, 3), Swin-T’s 1:1:3:1	79.36 ± 0.07	4.53
“patchify” stem	$7 \times 7$ stride-2 conv and max pool → $4 \times 4$ stride-4 conv	79.51 ± 0.18	4.42
depthwise conv	the bottleneck’s $3 \times 3$ conv made depthwise	78.28 ± 0.08	2.35
increase width	base width 64 → 96, Swin-T’s	80.50 ± 0.02	5.27
inverting dimensions	$1 \times 1$ expand ×4, depthwise $3 \times 3$, $1 \times 1$ reduce	80.64 ± 0.03	4.64
move up depthwise conv	depthwise conv first, then the two $1 \times 1$ convs	79.92 ± 0.08	4.07
kernel size → 5 / 7 / 9 / 11	depthwise kernel enlarged	80.35 / 80.57 / 80.57 / 80.47	4.10 / 4.15 / 4.21 / 4.29
ReLU → GELU	activation replaced	80.62 ± 0.14	4.15
fewer activations	one GELU per block, between the $1 \times 1$ convs	81.27 ± 0.06	4.15
fewer norms	one BN per block, before the $1 \times 1$ convs	81.41 ± 0.09	4.15
BN → LN	that BN replaced by LayerNorm	81.47 ± 0.09	4.46
separate downsampling	$2 \times 2$ stride-2 conv layers between stages (ConvNeXt-T)	81.97 ± 0.06	4.49
Swin-T	reference	81.30	4.50

The main paper’s Figure 2 shows the same sequence with accuracies rounded to one decimal. Its FLOPs for the BN → LN step are 4.2, not the 4.46 of Table 10; the example below counts this step and finds no change in multiply-adds from the step before.

The steps group into five themes (§2.2–§2.6):

• Macro design. Swin-T distributes blocks over stages as 1:1:3:1; ResNet-50’s (3, 4, 6, 3) was largely empirical, the authors write, and its heavy res4 stage was meant for detectors operating on $14 \times 14$ maps. Moving to (3, 3, 9, 3) also aligned the FLOPs with Swin-T. The stem became a $4 \times 4$ stride-4 non-overlapping convolution, Swin’s patch size; the change was worth 0.1 points (79.4% to 79.5%), and the authors conclude that ResNet’s stem may be replaced by a simpler patchify layer with similar results. The patch embedding article treats this stem against ViT’s and convolutional stems.
• ResNeXt-ify. ResNeXt’s guiding principle, in the authors’ summary, is to use more groups and expand the width. ConvNeXt takes the extreme case, a depthwise convolution with one group per channel, which the authors compare to the weighted sum in self-attention: both mix information only across space, per channel, while $1 \times 1$ convolutions mix only across channels. Making the $3 \times 3$ convolution depthwise cut the FLOPs to 2.35G and the accuracy to 78.3%; widening to Swin-T’s 96 channels brought 80.5% at 5.27G.
• Inverted bottleneck. A transformer’s MLP block is four times wider inside than at its input, the inverted-bottleneck shape that MobileNetV2 popularized in ConvNets, the authors note. Inverting the block (Figure 3b) raised the depthwise layer’s cost but lowered the network’s to 4.6G, which the authors attribute to the much cheaper $1 \times 1$ shortcut convolutions of the downsampling blocks; accuracy rose from 80.5% to 80.6%, and in the ResNet-200 regime from 81.9% to 82.6%.
• Large kernels. To afford large kernels, the depthwise convolution moved to the top of the block, before the expansion (Figure 3c), as attention comes before the MLP in a transformer block: the expensive spatial operation now runs on the narrow width and the dense $1 \times 1$ layers do the heavy lifting. This temporarily cost accuracy (79.9%). Kernels of 3, 5, 7, 9 and 11 were then tried; accuracy rose to 80.6% at $7 \times 7$ and saturated there, while the FLOPs stayed roughly the same. For the ResNet-200 regime the main text reports no further gain beyond $7 \times 7$, while the supplementary material writes that the gain seems to saturate at $5 \times 5$ instead, as its Table 11 shows (82.32% at $5 \times 5$, 82.30% at $7 \times 7$).
• Micro design. GELU replaced ReLU with unchanged accuracy (80.6%). Removing all activations but the one between the two $1 \times 1$ layers, as in a transformer MLP, gave 81.3%, practically Swin-T’s level; removing two of the three BN layers gave 81.4%. Replacing the remaining BN by LayerNorm gave 81.5%: the authors report no difficulty training with LN, although directly substituting LN for BN in the original ResNet is suboptimal. Finally, separate $2 \times 2$ stride-2 convolutions between stages, as in Swin, at first made training diverge; adding normalization layers wherever the resolution changes (after the stem, before each downsampling layer and after the final pooling) stabilized it and gave 82.0%, against Swin-T’s 81.3%.

The authors state that none of these design choices is new in the ConvNet literature; each had been studied separately, but not together. Two cautions apply to reading the table. First, some steps change accuracy by about as much as their seed-to-seed standard deviation (the patchify stem: 79.51 ± 0.18 after 79.36 ± 0.07), so in our reading those steps show that a change does no harm rather than that it helps. Second, steps were adopted in a fixed order and each was judged in the network built by the steps before it; the depthwise convolution moved up only because an inverted bottleneck already existed. The ResNet-200 regime (Table 11) ran a similar sequence, whose stage-ratio step also widened the network from 64 to 84 channels and whose width was later raised to Swin-B’s 128 (supplementary §C); it gained more from inverting dimensions (+0.79 against +0.14 points) and from fewer normalizations (+0.46 against +0.14).

Stem

The stem is a $4 \times 4$ convolution with stride 4 from 3 to $C_1$ channels, with a bias, followed by LayerNorm over the channels (official code, models/convnext.py). For ConvNeXt-T it has $4^2 \cdot 3 \cdot 96 = 4{,}608$ weights.

ConvNeXt Block

For an input map $\mathbf{x}$ with $C$ channels, the block computes

$$\mathbf{y} = \mathbf{x} + \boldsymbol{\gamma} \odot W_2\,\operatorname{GELU}\!\left(W_1\,\operatorname{LN}\!\left(\operatorname{DWConv}_{7 \times 7}(\mathbf{x})\right)\right),$$

where $\operatorname{DWConv}_{7 \times 7}$ is a depthwise $7 \times 7$ convolution with padding 3, LN normalizes the $C$ channels at each position, $W_1$ maps $C$ to $4C$ channels and $W_2$ maps $4C$ back to $C$ at each position, and $\boldsymbol{\gamma} \in \mathbb{R}^C$ is a learned per-channel scale, the layer scale that the paper cites to Touvron et al.’s CaiT (“Going deeper with image transformers”), initialized to $10^{-6}$ in the official code. The block contains one normalization and one activation, and nothing follows the addition. The official implementation permutes the map to channels-last after the depthwise convolution and implements the two $1 \times 1$ convolutions as linear layers, which its docstring says is slightly faster in PyTorch; the layer normalization article covers this channels-last LN and how it differs from an LN over all of a map’s channels and positions. During training a stochastic-depth layer drops the residual branch at random.

Per position, the block costs $49C$ multiply-adds in the depthwise convolution and $8C^2$ in the two $1 \times 1$ layers. At $C = 96$ the $1 \times 1$ layers take 94% of the block’s multiply-adds; at $C = 768$, 99%. In our reading, this is what makes the $7 \times 7$ kernel affordable: as the depthwise separable convolution article shows for one layer, the kernel size enters only the small depthwise term.

Downsampling Layers

Between stages, a LayerNorm followed by a $2 \times 2$ convolution with stride 2 halves the resolution and doubles the width. Inside the stages the blocks keep resolution and width, so no block needs a projection shortcut. After the last stage, global average pooling, a final LayerNorm and a linear layer give the class scores.

Isotropic ConvNeXt

To test the block outside a hierarchy, the authors also built ViT-style isotropic networks: a patchify stem to a $14 \times 14$ map and a stack of ConvNeXt blocks at a single width, 384, 768 or 1024 channels as in ViT-S, B and L, with 18, 18 and 36 blocks to match parameters and FLOPs (§3.2). They train with 50 warmup epochs and, for S and B, without layer scale (supplementary §A.1). The official repository provides ImageNet-1K checkpoints for all three.

Training

ConvNeXt is trained as a supervised ImageNet classifier with label smoothing; the paper does not write out its loss, and what is distinctive is the recipe (supplementary Table 5). On ImageNet-1K: AdamW with base learning rate $4 \times 10^{-3}$, weight decay 0.05 and batch size 4,096; 300 epochs with 20 linear warmup epochs and cosine decay; RandAugment (9, 0.5), Mixup 0.8, CutMix 1.0, Random Erasing 0.25, label smoothing 0.1; stochastic depth 0.1, 0.4, 0.5 and 0.5 for T, S, B and L; layer scale initialized to $10^{-6}$; truncated-normal weight initialization, listed in Table 5 as “trunc. normal (0.2)”, while the official code uses a standard deviation of 0.02; and an exponential moving average of the weights (0.9999). The roadmap experiments used the same settings with the moving average disabled, which the authors found severely hurts models with BatchNorm layers. The learning rate schedule article compares this schedule with other recipes; the decay settings appear beside other models’ in the weight decay article.

ImageNet-22K pre-training (about 14 million images in 21,841 classes) uses the same optimizer for 90 epochs with 5 warmup epochs and no moving average. Fine-tuning on ImageNet-1K runs for 30 epochs with AdamW at $5 \times 10^{-5}$, layer-wise learning-rate decay (0.7 or 0.8, with every three consecutive blocks forming a group) and no Mixup or CutMix (supplementary Table 6).

Inference

ConvNeXt takes an RGB image and returns class scores; as a backbone it returns the four stage maps at strides 4 to 32. Because it is fully convolutional, the same weights run at other input sizes: ConvNeXt-T costs 4.5 GFLOPs at $224^2$ and 13.1 GFLOPs at $384^2$ (Table 1). At $384^2$, a crop ratio of 1.0 is used at test time instead of 0.875 (supplementary §A.2).

The authors measured inference throughput against Swin Transformers with similar FLOPs. On a V100 GPU ConvNeXt-T processed 774.7 images per second against Swin-T’s 757.9, and ConvNeXt-B at $384^2$ processed 95.7 against Swin-B’s 85.1 (Table 1). On an A100 GPU with TF32 and the channels-last memory layout of PyTorch 1.10, ConvNeXt-T processed 1,943.5 images per second against 1,325.6 (+47%), and the advantage reached about 49% for ConvNeXt-S (supplementary Table 12). The authors note that under similar FLOPs, models with depthwise convolutions are known to be slower and to use more memory than ConvNets with only dense convolutions, and report that ConvNeXt’s throughput nevertheless matches or exceeds Swin’s; training Cascade Mask R-CNN with a ConvNeXt-B backbone used 17.4 GB of peak memory at a per-GPU batch size of 2, against 18.5 GB for Swin-B.

Performance

ImageNet classification (ImageNet-1K validation top-1, single crop; Table 1). Trained on ImageNet-1K at $224^2$, ConvNeXt-T, S and B reached 82.1%, 83.1% and 83.8%, against 81.3%, 83.0% and 83.5% for Swin-T, S and B at the same FLOPs. At $384^2$, ConvNeXt-B reached 85.1% against Swin-B’s 84.5%, and ConvNeXt-L 85.5%. With ImageNet-22K pre-training and ImageNet-1K fine-tuning, ConvNeXt-B, L and XL reached 85.8%, 86.6% and 87.0% at $224^2$ and 86.8%, 87.5% and 87.8% at $384^2$; the 87.8% of ConvNeXt-XL is the paper’s headline figure. EfficientNetV2-L remained ahead among ImageNet-1K-trained models (85.7% at $480^2$), and the authors note that ConvNeXt overtakes EfficientNetV2 with ImageNet-22K pre-training (87.3% for EfficientNetV2-XL).

Isotropic networks (Table 2). Isotropic ConvNeXt-S, B and L reached 79.7%, 82.0% and 82.6% against 79.8%, 81.8% and 82.6% for ViT-S, B and L with DeiT’s and MAE’s supervised recipes. The repository’s README (at commit 06f7b05, and unchanged at the latest commit, 048efce) lists the isotropic ConvNeXt-S checkpoint at 78.7%, not Table 2’s 79.7%.

Object detection and segmentation on COCO (Table 3, 3× schedule with multi-scale training; the paper’s text does not name the evaluation split). With Mask R-CNN, ConvNeXt-T reached 46.2 box AP and 41.7 mask AP against Swin-T’s 46.0 and 41.6. With Cascade Mask R-CNN, ConvNeXt-T matched Swin-T at 50.4 box AP and 43.7 mask AP (ResNet-50: 46.3 and 40.1), and with ImageNet-22K pre-training ConvNeXt-B reached 54.0 and 46.9 against Swin-B’s 53.0 and 45.8; ConvNeXt-XL reached 55.2 and 47.7.

Semantic segmentation on ADE20K (Table 4, UperNet, 160K iterations, validation mIoU with multi-scale testing). ConvNeXt-T reached 46.7 against Swin-T’s 45.8, and ConvNeXt-B with ImageNet-22K pre-training at $640^2$ reached 53.1 against Swin-B’s 51.7; ConvNeXt-XL reached 54.0. With single-scale testing, ConvNeXt-T reached 46.0 (supplementary Table 7).

Robustness (supplementary Table 8). Without specialized modules or extra fine-tuning, ConvNeXt-T had a mean corruption error of 53.2 on ImageNet-C against ResNet-50’s 76.7 and Swin-T’s 62.0, and ConvNeXt-XL with ImageNet-22K data reached 69.3%, 68.2% and 55.0% on ImageNet-A, ImageNet-R and ImageNet-Sketch.

Strengths

• Standard modules only. Convolutions, LayerNorm, GELU and linear layers: no attention, shifted windows or relative position biases, so the network runs wherever convolutions are optimized and slides over inputs of any size.
• Transformer-level accuracy at matched cost. In the paper’s comparisons it matches or exceeds Swin at similar FLOPs in ImageNet top-1 accuracy, COCO box and mask AP and ADE20K mIoU.
• A drop-in hierarchical backbone. Its four stage maps at strides 4 to 32 match ResNet’s and Swin’s, so detectors and segmenters built for those backbones accept it; the authors used Mask R-CNN, Cascade Mask R-CNN and UperNet with Swin’s training settings.
• A controlled study. The roadmap separates the effects of recipe and architecture in one compute regime, which system-level comparisons had confounded.

Limitations

• Tasks that may suit transformers. In their limitations section (supplementary §F) the authors note that transformers may be more flexible for multi-modal learning, where cross-attention models interactions across many modalities, and for tasks requiring discretized, sparse or structured outputs.
• FLOPs are not speed. Under similar FLOPs, models with depthwise convolutions are known to be slower and to use more memory than dense ConvNets, as the authors themselves note, and their throughput advantage was measured on V100 and A100 GPUs with a specific memory layout; the same FLOPs can run at different speeds on other hardware.
• One recipe, one order. Every roadmap step was judged with a recipe tuned for transformers and in a fixed order; in our reading, a different order or recipe might have kept different changes. The stage ratio, the authors note, is likely not optimal.
• Supervised pre-training. Masked image modeling, which improved ViTs, did not improve ConvNeXt as such; the V2 paper reports that a ConvNeXt-B pre-trained with its masked autoencoder but without architectural change fine-tuned to 83.7%, no better than the supervised 83.8%.

Historical Importance

ConvNeXt appeared when hierarchical vision transformers such as Swin had begun, in the authors’ description, to overtake ConvNets as the favored choice for generic vision backbones. The paper’s first finding was that a significant portion of the difference between ConvNets and vision transformers may be due to training techniques; in our summary, its second was that the rest of the difference closed with design choices that need no attention. The roadmap was the evidence, with every step documented with its cost and accuracy. Later work on this site uses ConvNeXt as a backbone or student: DINOv3 distills its large ViT teacher into smaller models, with ConvNeXt variants among its released backbones, FC-CLIP builds an open-vocabulary segmentation model on a frozen convolutional CLIP with a ConvNeXt-Large image encoder, and BEVDepth reports results with a ConvNeXt backbone in bird’s-eye-view perception. The convolutional neural network and deep learning articles place it in the longer history of architectures.

Variants

ConvNeXt V2 (2023)

Woo, Debnath, Hu, Chen, Liu, Kweon and Xie asked whether ConvNeXt could benefit from masked autoencoders as ViTs had, and found that simply combining the two gave subpar results. Their answer has two parts.

Fully convolutional masked autoencoder (FCMAE). Of the input’s $32 \times 32$ patches, 60% are removed at random; the mask is generated at the last stage’s resolution and upsampled to the input. A transformer encoder can drop the masked tokens, but a convolution slides over them and leaks information from the masked region. FCMAE therefore treats the masked image as a sparse array of pixels and replaces the encoder’s convolutions by submanifold sparse convolutions that compute only at visible positions; at fine-tuning they become ordinary dense convolutions again. The decoder is a single ConvNeXt block of width 512, and the loss is the mean squared error between the reconstruction and the patch-wise normalized image on the masked patches only, as in MAE (§3). Without sparse convolution, a ConvNeXt-B pre-trained this way fine-tuned to 79.3%; with it, to 83.7%.

Global response normalization (GRN). Visualizing the FCMAE-trained ConvNeXt V1, the authors found feature collapse: many dead or saturated channels in the dimension-expansion MLP layer, with redundant activations across channels. GRN works on that layer’s output $X \in \mathbb{R}^{H \times W \times C}$ in three steps (§4, Eqs. 1–3): it aggregates each channel by its L2 norm over space, normalizes these norms by dividing by their sum over channels, and rescales each channel by the result:

$$g_i = \lVert X_i \rVert_2, \qquad n_i = \frac{g_i}{\sum_{j=1}^{C} g_j}, \qquad X_i' = \gamma\, X_i\, n_i + \beta + X_i .$$

The authors also scale $n_i$ by the channel count $C$ in practice (a footnote), which their Algorithm 1 implements by dividing by the mean of the norms instead of their sum. The learnable $\gamma$ and $\beta$ start at zero, so the layer begins as the identity through its residual term. GRN goes after the expansion layer in every block, and layer scale is removed as redundant. In the paper’s ablations on ConvNeXt-B (Table 2), GRN gave 84.6% against 83.8% for LayerNorm, 80.5% for BatchNorm and 83.2% for local response normalization in its place, and outperformed squeeze-and-excitation (84.4%) and CBAM (84.5%) without their extra parameters (89M against 109M). The supplementary material calls GRN “Global Relation Network” in its component analysis (§D), while the main paper names it global response normalization.

Results. Neither change alone helped much: ConvNeXt V1-B reached 83.8% supervised and 83.7% with FCMAE, V2-B supervised 84.3%, and V2-B with FCMAE 84.6% (Table 3, 800 pre-training epochs). The V2 family spans eight sizes, from Atto (3.7M parameters, 76.7% with FCMAE) through Femto, Pico and Nano, the small configurations designed originally in Wightman’s timm library, to Tiny, Base, Large and a new Huge model. ConvNeXt V2-H, pre-trained with FCMAE, fine-tuned on ImageNet-22K and then on ImageNet-1K, reached 88.9% top-1 at $512^2$ (Table 5). The paper gives Huge’s size as 650M in its abstract and text and as 659M in its tables; the supplementary material gives 659M in its list of configurations (§A.1) and 660M in its Tables 7 and 8. The same supplementary table lists ConvNeXt V1-T at 4.47 GFLOPs, against the 4.5G of the V1 paper’s Tables 1 and 9 and the 4.49 of its Table 10.

Implementations

The official implementation is the PyTorch code in the facebookresearch/ConvNeXt repository, released under the MIT licence, with ImageNet-1K and ImageNet-22K checkpoints, training and fine-tuning scripts, and detection and segmentation code built on MMDetection and MMSegmentation. As of November 2026 its latest commit dates from July 2022. ConvNeXt V2’s code is in the facebookresearch/ConvNeXt-V2 repository named in its paper.

The roadmap counted. The example (PyTorch 2.14.1, four threads) rebuilds each network of Table 10 with random weights, from a ResNet-50 in the torchvision layout (stride in the $3 \times 3$ convolution) to the official ConvNeXt-T structure, and runs one synthetic $224 \times 224$ image (stripes and a disk) through each. For every network it prints the parameter count, the multiply-adds counted by PyTorch’s FLOP counter (two FLOPs per multiply-add; convolutions and matrix products only, not normalizations, activations, pooling or additions), the multiply-adds of each kind of layer from forward hooks, and the activations and normalizations in one block of stage 3. The counts do not depend on the weights or the image content.

The paper’s text leaves two choices open, and both affect the count: whether the stem widens to 96 channels with the rest of the network at the “increase width” step or stays at ResNet’s 64, and whether the hidden width of the first, downsampling block of each stage is four times its output or its input width once blocks are inverted. The code builds all four combinations, prints the full table for one of them and, for each combination, how many of the 14 steps from ResNet-50 to “fewer norms” round to Table 10. Before the last step, downsampling happens as in ResNet, which the paper describes: in the first block of each stage, with a stride-2 spatial convolution and a $1 \times 1$ stride-2 projection shortcut. Where the stride sits in the inverted and moved blocks is not stated; our code keeps it in the spatial (depthwise) convolution, a convention that does affect the multiply-adds. Other details are our own conventions where the paper is silent and do not change the multiply-adds: the projection shortcut has a normalization, intermediate networks have no convolution biases, and the patchify stem keeps a normalization and an activation. The intermediate parameter counts therefore follow our conventions and have no counterpart in the paper. The script takes about 30 seconds to a minute on our machine (approximate, as agents share it) and prints the same output on every run.

The first part defines a residual block for every step before the last, with switches for the layout, kernel size, depthwise or dense spatial convolution, activation, and the number and kind of normalizations:

import torch
from torch import nn
from torch.utils.flop_counter import FlopCounterMode

torch.manual_seed(0)
torch.set_num_threads(4)

class LayerNorm2d(nn.LayerNorm):
    """LayerNorm over the channels at every position of an (N, C, H, W) map."""
    def forward(self, x):
        return super().forward(x.permute(0, 2, 3, 1)).permute(0, 3, 1, 2)

def norm(kind, c):
    return nn.BatchNorm2d(c) if kind == "BN" else LayerNorm2d(c, eps=1e-6)

class Block(nn.Module):
    """One residual block at a step of the roadmap.
    layout: 'bottleneck' 1x1 -> kxk -> 1x1 (ResNet; groups=width makes the kxk depthwise),
            'inverted'   1x1 (expand 4x) -> depthwise kxk -> 1x1 (Fig. 3b),
            'moved'      depthwise kxk -> 1x1 (expand 4x) -> 1x1 (Fig. 3c).
    acts: activations after every conv except the last, plus one after the addition ('all'),
          or a single one between the two 1x1 convs ('one'). norms: after every conv ('all') or
          only after the spatial conv ('one'). hidden: 4 x output or 4 x input channels."""
    def __init__(self, c_in, width, stride, layout, k, depthwise, act, acts, norms, nk, hidden):
        super().__init__()
        A = nn.ReLU if act == "ReLU" else nn.GELU
        if layout == "bottleneck":
            c_out, mid = 4 * width, width
            convs = [nn.Conv2d(c_in, mid, 1, bias=False),
                     nn.Conv2d(mid, mid, k, stride, k // 2, groups=mid if depthwise else 1, bias=False),
                     nn.Conv2d(mid, c_out, 1, bias=False)]
            spatial = 1
        else:
            c_out = width
            e = 4 * (c_out if hidden == "out" else c_in)
            if layout == "inverted":
                convs = [nn.Conv2d(c_in, e, 1, bias=False),
                         nn.Conv2d(e, e, k, stride, k // 2, groups=e, bias=False),
                         nn.Conv2d(e, c_out, 1, bias=False)]
                spatial = 1
            else:
                convs = [nn.Conv2d(c_in, c_in, k, stride, k // 2, groups=c_in, bias=False),
                         nn.Conv2d(c_in, e, 1, bias=False),
                         nn.Conv2d(e, c_out, 1, bias=False)]
                spatial = 0
        layers = []
        for i, cv in enumerate(convs):
            layers.append(cv)
            if norms == "all" or i == spatial:
                layers.append(norm(nk, cv.out_channels))
            if (acts == "all" and i < 2) or (acts == "one" and i == 1):
                layers.append(A())
        self.f = nn.Sequential(*layers)
        self.shortcut = nn.Identity() if stride == 1 and c_in == c_out else nn.Sequential(
            nn.Conv2d(c_in, c_out, 1, stride, bias=False), norm(nk, c_out))
        self.post = A() if acts == "all" else nn.Identity()
        self.c_out = c_out
    def forward(self, x):
        return self.post(self.f(x) + self.shortcut(x))

The final block and the network follow the official models/convnext.py; the network class also builds every intermediate step:

class ConvNeXtBlock(nn.Module):
    """The final block, as in the official code: depthwise 7x7, LN, Linear, GELU, Linear, layer scale."""
    def __init__(self, dim):
        super().__init__()
        self.dwconv = nn.Conv2d(dim, dim, 7, padding=3, groups=dim)
        self.norm = nn.LayerNorm(dim, eps=1e-6)
        self.pwconv1, self.act, self.pwconv2 = nn.Linear(dim, 4 * dim), nn.GELU(), nn.Linear(4 * dim, dim)
        self.gamma = nn.Parameter(1e-6 * torch.ones(dim))
        self.c_out = dim
    def forward(self, x):
        y = self.dwconv(x).permute(0, 2, 3, 1)
        y = self.gamma * self.pwconv2(self.act(self.pwconv1(self.norm(y))))
        return x + y.permute(0, 3, 1, 2)

class Net(nn.Module):
    def __init__(self, depths=(3, 4, 6, 3), stem="resnet", stem_width=64, base=64, layout="bottleneck",
                 k=3, depthwise=False, act="ReLU", acts="all", norms="all", nk="BN", hidden="out",
                 final=False):
        super().__init__()
        if stem == "resnet":
            self.stem = nn.Sequential(nn.Conv2d(3, stem_width, 7, 2, 3, bias=False), norm(nk, stem_width),
                                      nn.ReLU(), nn.MaxPool2d(3, 2, 1))
        elif final:
            self.stem = nn.Sequential(nn.Conv2d(3, stem_width, 4, 4), LayerNorm2d(stem_width, eps=1e-6))
        else:
            self.stem = nn.Sequential(nn.Conv2d(3, stem_width, 4, 4, bias=False), norm(nk, stem_width),
                                      nn.ReLU() if act == "ReLU" else nn.GELU())
        c, stages = stem_width, []
        for i, n in enumerate(depths):
            w, blocks = base * 2 ** i, []
            if final and i > 0:                                   # separate downsampling layer
                blocks.append(nn.Sequential(LayerNorm2d(c, eps=1e-6), nn.Conv2d(c, w, 2, 2)))
                c = w
            for j in range(n):
                if final:
                    b = ConvNeXtBlock(c)
                else:
                    b = Block(c, w, 2 if (j == 0 and i > 0) else 1, layout, k, depthwise, act, acts, norms, nk, hidden)
                blocks.append(b)
                c = b.c_out
            stages.append(nn.Sequential(*blocks))
        self.stages = nn.Sequential(*stages)
        self.final_norm = nn.LayerNorm(c, eps=1e-6) if final else nn.Identity()
        self.head = nn.Linear(c, 1000)
    def forward(self, x):
        x = self.stages(self.stem(x))
        self.last_map = tuple(x.shape[-2:])
        return self.head(self.final_norm(x.mean((2, 3))))

The measurement assigns each convolution or linear layer to a role by its name and kernel size and checks that the hooks and the FLOP counter agree:

def role(name, m):
    if name.startswith("stem"):
        return "stem"
    if name.startswith("head"):
        return "head"
    if "shortcut" in name or isinstance(m, nn.Conv2d) and m.kernel_size == (2, 2):
        return "short/down"
    if isinstance(m, nn.Conv2d) and m.kernel_size[0] > 1:
        return "kxk"
    return "1x1"

def measure(net, image):
    """Parameters; multiply-adds from PyTorch's FLOP counter (2 FLOPs per multiply-add);
    multiply-adds per role from forward hooks; activations and norms in one stage-3 block."""
    parts = {}
    def hook(name):
        def f(m, inp, out):
            per_out = (m.kernel_size[0] * m.kernel_size[1] * m.in_channels // m.groups
                       if isinstance(m, nn.Conv2d) else m.in_features)
            parts[role(name, m)] = parts.get(role(name, m), 0) + out.numel() * per_out
        return f
    hs = [m.register_forward_hook(hook(n)) for n, m in net.named_modules() if isinstance(m, (nn.Conv2d, nn.Linear))]
    with torch.no_grad(), FlopCounterMode(display=False) as fc:
        logits = net.eval()(image)
    for h in hs:
        h.remove()
    blk = net.stages[2][-1]
    acts = sum(isinstance(m, (nn.ReLU, nn.GELU)) for m in blk.modules())
    norms = [type(m).__name__ for m in blk.modules() if isinstance(m, (nn.BatchNorm2d, nn.LayerNorm))]
    act_name = next((type(m).__name__ for m in blk.modules() if isinstance(m, (nn.ReLU, nn.GELU))), "-")
    ok = logits.shape == (1, 1000) and bool(torch.isfinite(logits).all()) and net.last_map == (7, 7)
    return (sum(p.numel() for p in net.parameters()), fc.get_total_flops() // 2, parts,
            f"{act_name}x{acts} {norms[0].replace('BatchNorm2d', 'BN').replace('LayerNorm2d', 'LN').replace('LayerNorm', 'LN')}x{len(norms)}", ok)

The last part lists the steps with their Table 10 GFLOPs, runs all four readings and prints the results:

# The roadmap, ResNet-50 regime (Liu et al., 2022, Sec. 2; GFLOPs from supplementary Table 10).
def roadmap(stem_width, hidden):
    s = [("ResNet-50", dict(), 4.09)]
    c = dict(depths=(3, 3, 9, 3));                       s.append(("stage ratio", c, 4.53))
    c = dict(c, stem="patchify");                        s.append(("patchify stem", c, 4.42))
    c = dict(c, depthwise=True);                         s.append(("depthwise conv", c, 2.35))
    c = dict(c, base=96, stem_width=stem_width);         s.append(("increase width", c, 5.27))
    c = dict(c, layout="inverted", hidden=hidden);       s.append(("inverting dims", c, 4.64))
    c = dict(c, layout="moved");                         s.append(("move up dw conv", c, 4.07))
    for kk, ref in ((5, 4.10), (7, 4.15), (9, 4.21), (11, 4.29)):
        s.append((f"kernel {kk}", dict(c, k=kk), ref))
    c = dict(c, k=7)                                     # 7x7 kept
    c = dict(c, act="GELU");                             s.append(("ReLU -> GELU", c, 4.15))
    c = dict(c, acts="one");                             s.append(("fewer activations", c, 4.15))
    c = dict(c, norms="one");                            s.append(("fewer norms", c, 4.15))
    c = dict(c, nk="LN");                                s.append(("BN -> LN", c, 4.46))
    s.append(("sep. downsampling", dict(depths=(3, 3, 9, 3), stem="patchify", stem_width=96, base=96,
                                         final=True), 4.49))
    return s

# A synthetic 224x224 RGB test image: a bright disk on vertical stripes.
yy, xx = torch.meshgrid(torch.arange(224.0), torch.arange(224.0), indexing="ij")
image = torch.stack([(xx % 16 < 8).float(), ((yy - 112) ** 2 + (xx - 112) ** 2 < 70 ** 2).float(),
                     0.5 * torch.ones(224, 224)])[None]

ROLES = ["stem", "kxk", "1x1", "short/down", "head"]
all_ok, results = True, {}
for sw, hid in ((96, "out"), (64, "out"), (96, "in"), (64, "in")):
    results[sw, hid] = []
    for name, cfg, ref in roadmap(sw, hid):
        params, macs, parts, desc, ok = measure(Net(**cfg), image)
        assert sum(parts.values()) == macs            # hooks and FLOP counter agree
        all_ok &= ok
        results[sw, hid].append((name, params, macs, parts, desc, ref))

print("Stem width 96 from 'increase width' on; hidden width 4 x output channels")
print(f"{'step':18s} {'params':>11s} {'block (stage 3)':>16s} " + "".join(f"{r:>11s}" for r in ROLES)
      + "   GMACs  Table 10")
for name, params, macs, parts, desc, ref in results[96, "out"]:
    print(f"{name:18s} {params:11,d} {desc:>16s} " + "".join(f"{parts.get(r, 0) / 1e9:11.4f}" for r in ROLES)
          + f"   {macs / 1e9:5.3f}   {ref:.2f}{'' if round(macs / 1e9, 2) == ref else '  differs'}")

print("\nSteps ResNet-50 .. fewer norms whose GMACs round to Table 10, by reading of the open choices:")
for (sw, hid), rows in results.items():
    rr = [r for r in rows if r[0] not in ("BN -> LN", "sep. downsampling")]
    hits = sum(round(r[2] / 1e9, 2) == r[5] for r in rr)
    worst = max(abs(r[2] / 1e9 - r[5]) for r in rr)
    print(f"  stem width {sw}, hidden 4 x {hid:3s}: {hits}/{len(rr)}, largest difference {worst:.3f} G")
print("every network: logits (1, 1000), finite, last feature map 7x7:", bool(all_ok))

Output:

Stem width 96 from 'increase width' on; hidden width 4 x output channels
step                    params  block (stage 3)        stem        kxk        1x1 short/down       head   GMACs  Table 10
ResNet-50           25,557,032      ReLUx3 BNx3      0.1180     1.8497     1.7598     0.3597     0.0020   4.089   4.09
stage ratio         28,628,520      ReLUx3 BNx3      0.1180     2.0809     1.9653     0.3597     0.0020   4.526   4.53
patchify stem       28,622,184      ReLUx3 BNx3      0.0096     2.0809     1.9653     0.3597     0.0020   4.418   4.42
depthwise conv      15,722,664      ReLUx3 BNx3      0.0096     0.0129     1.9653     0.3597     0.0020   2.350   2.35
increase width      33,761,032      ReLUx3 BNx3      0.0145     0.0193     4.4219     0.8092     0.0031   5.268   5.27
inverting dims      25,851,496      ReLUx3 BNx3      0.0145     0.0772     4.5086     0.0434     0.0008   4.644   4.64
move up dw conv     25,625,512      ReLUx3 BNx3      0.0145     0.0181     3.9884     0.0434     0.0008   4.065   4.07
kernel 5            25,720,744      ReLUx3 BNx3      0.0145     0.0503     3.9884     0.0434     0.0008   4.097   4.10
kernel 7            25,863,592      ReLUx3 BNx3      0.0145     0.0987     3.9884     0.0434     0.0008   4.146   4.15
kernel 9            26,054,056      ReLUx3 BNx3      0.0145     0.1631     3.9884     0.0434     0.0008   4.210   4.21
kernel 11           26,292,136      ReLUx3 BNx3      0.0145     0.2436     3.9884     0.0434     0.0008   4.291   4.29
ReLU -> GELU        25,863,592      GELUx3 BNx3      0.0145     0.0987     3.9884     0.0434     0.0008   4.146   4.15
fewer activations   25,863,592      GELUx1 BNx3      0.0145     0.0987     3.9884     0.0434     0.0008   4.146   4.15
fewer norms         25,797,352      GELUx1 BNx1      0.0145     0.0987     3.9884     0.0434     0.0008   4.146   4.15
BN -> LN            25,797,352      GELUx1 LNx1      0.0145     0.0987     3.9884     0.0434     0.0008   4.146   4.46  differs
sep. downsampling   28,589,128      GELUx1 LNx1      0.0145     0.1051     4.1618     0.1734     0.0008   4.456   4.49  differs

Steps ResNet-50 .. fewer norms whose GMACs round to Table 10, by reading of the open choices:
  stem width 96, hidden 4 x out: 14/14, largest difference 0.005 G
  stem width 64, hidden 4 x out: 4/14, largest difference 0.055 G
  stem width 96, hidden 4 x in : 5/14, largest difference 0.521 G
  stem width 64, hidden 4 x in : 4/14, largest difference 0.612 G
every network: logits (1, 1000), finite, last feature map 7x7: True

Only one reading reproduces Table 10 at every step from ResNet-50 to “fewer norms”: a stem that widens to 96 channels with the network, and a hidden width four times the output width in the downsampling blocks. The other three miss most steps, by up to 0.6 G. We infer from this that the authors’ networks made these two choices, given the stride placement above; the paper does not state them. Under this reading every count rounds to the table’s two decimals, which in our reading suggests that Table 10’s FLOPs for these steps are, or are very close to, the multiply-adds of convolutions and linear layers.

The role columns show where the cost moves. Making the $3 \times 3$ convolutions depthwise removes almost all of their 2.08 G, and the next step, widening the network by half (64 to 96 base channels) as ResNeXt’s principle suggests, spends more than the saving: the total rises above the cost before the depthwise step (5.268 G against 4.418 G). Inverting the block quadruples the depthwise layers’ cost (0.019 to 0.077 G), but the shortcut projections of the downsampling blocks fall from 0.81 to 0.04 G because they now connect narrow inputs and outputs: the drop in the total (0.62 G) is smaller than the drop in the shortcuts alone, as the paper’s explanation requires. Moving the depthwise convolution up cuts its cost to about a quarter again, which leaves room for the larger kernels, and from $3 \times 3$ to $11 \times 11$ the depthwise layers grow from 0.02 to 0.24 G while the $1 \times 1$ layers stay at 3.99 G. The GELU, activation, normalization and LayerNorm steps change no multiply-adds, as the block column shows they change only those layers.

Two figures in the paper differ from the count. For the BN → LN step, Table 10 gives 4.46 GFLOPs, while the count is unchanged at 4.146 G and the paper’s own Figure 2 gives 4.2. In our reading the 4.46 cannot come from swapping a normalization layer, which has no multiply-adds of this kind; it is close to our count for the next network, with separate downsampling layers (4.456 G), but the paper does not say whether the row is a misprint. In the ResNet-200 regime, Table 11 gives the BN → LN step the same 14.81 GFLOPs as the step before it. For that final network the count is 4.456 G against Table 10’s 4.49, and it rounds to the 4.5G of Tables 1 and 9. The paper does not state which operations its FLOP figures include; in our reading the remaining 0.03 G could come from counting operations, such as normalizations, that our counter leaves out, but the paper gives no way to check this. The separate downsampling step also shows where most of its 0.31 G come from: the $2 \times 2$ downsampling convolutions (0.17 G against 0.04 G of shortcuts) and the first block of each stage, whose first $1 \times 1$ layer now starts from the doubled width produced by the downsampling layer instead of from the previous stage’s width. The final parameter count, 28,589,128, matches Table 9’s $28.6 \times 10^6$, and ResNet-50’s 25,557,032 matches torchvision’s, as the ResNet article’s example also finds.