ResNet

ResNet is the family of residual convolutional networks that Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun of Microsoft Research introduced for image classification in a December 2015 technical report, published at CVPR 2016. Every member has the same plan: a strided $7 \times 7$ convolution and a max pool, four stages of residual blocks that halve the resolution and double the width from one stage to the next, global average pooling and one fully connected layer. Depth is set by the number of blocks per stage and by the block type, a two-layer basic block for ResNet-18 and ResNet-34 and a three-layer bottleneck block for ResNet-50, -101 and -152. The publication itself, its experiments and its awards are covered in the ResNet paper article; this article covers the architecture family as it is used today, its main variants, and how much of a ResNet’s reported accuracy comes from the way it is trained.

Problem Addressed

By 2015, the deepest plain networks trained worse than shallower ones: adding layers to a VGG-style stack raised the training error. This degradation problem is distinct from vanishing gradients, as the article on vanishing and exploding gradients explains. Residual blocks removed that obstacle, and the design question became how to build a deep network that was still affordable. He et al. kept VGG’s two design rules for the plain stack (see the paper article), moved downsampling into stride-2 convolutions, replaced VGG’s hidden fully connected layers by global average pooling before a single classifier layer, and used bottleneck blocks for 50 layers and more. The result was a family that went from 1.8 to 11.3 billion multiply-adds, all below VGG-16, and that other systems could use unchanged as a feature extractor.

Architecture

A ResNet maps a $3 \times 224 \times 224$ image to 1000 class scores in three parts: a stem that reduces the resolution by 4, four stages named conv2_x to conv5_x after Table 1 of He et al. (2016), and a head. Throughout this section, tensors are $(N, C, H, W)$ as in PyTorch, and sizes are for a $224 \times 224$ input.

Stem

A $7 \times 7$ convolution with 64 filters and stride 2 reduces the image to $64 \times 112 \times 112$, and a $3 \times 3$ max pool with stride 2 to $64 \times 56 \times 56$. The stem convolution has only $7^2 \cdot 3 \cdot 64 = 9{,}408$ weights, but it runs at the highest resolution: in the example below it accounts for 2.9% of ResNet-50’s multiply-adds. Later variants changed the stem without changing the rest. ResNet-C (He, T. et al., 2019) replaces the $7 \times 7$ convolution by three $3 \times 3$ convolutions, an idea they attribute to Inception-v2, and ConvNeXt (Liu et al., 2022) uses a $4 \times 4$ convolution with stride 4, like a vision transformer’s patch embedding, which in their modernized ResNet-50 changed ImageNet top-1 accuracy from 79.4% to 79.5%.

Stages and Output Strides

Each stage is a sequence of identical blocks. The first block of conv3_x, conv4_x and conv5_x has stride 2, so the four stages produce maps at $56 \times 56$, $28 \times 28$, $14 \times 14$ and $7 \times 7$, that is at output strides 4, 8, 16 and 32 relative to the input:

Stage	Output size	Output stride	Basic block (channels)	Bottleneck block (inner → output channels)
conv2_x	$56 \times 56$	4	64	64 → 256
conv3_x	$28 \times 28$	8	128	128 → 512
conv4_x	$14 \times 14$	16	256	256 → 1024
conv5_x	$7 \times 7$	32	512	512 → 2048

Everything before the head is convolutional, so the same weights accept other input sizes: the stage outputs scale with the input, and only the global average pool fixes the length of the feature vector. Detection and segmentation systems use this hierarchy directly (see Historical Importance). The final stride of 32 can be reduced by replacing the stride of the last stages with dilation, as DeepLab does; TorchVision exposes this as replace_stride_with_dilation.

Basic and Bottleneck Blocks

The basic block is two $3 \times 3$ convolutions, each followed by batch normalization, with a ReLU between them and a ReLU after the addition of the shortcut. The bottleneck block is a $1 \times 1$ convolution that reduces the channels to a width $w$, a $3 \times 3$ convolution at width $w$, and a $1 \times 1$ convolution that expands to $4w$, again with batch normalization after each convolution and the last ReLU after the addition. The order conv → BN → ReLU, with the final ReLU after the sum, is the original post-activation design; the pre-activation variant below reverses it.

The two blocks cost about the same but process maps of different widths. A basic block on 64 channels has $2 \cdot 3^2 \cdot 64 \cdot 64 = 73{,}728$ weights; a bottleneck block on 256 channels with $w = 64$ has $256 \cdot 64 + 3^2 \cdot 64 \cdot 64 + 64 \cdot 256 = 69{,}632$, the “≈70k” that Xie et al. (2017) compute for the same block. A bottleneck network therefore carries four times wider feature maps between blocks at a similar cost: ResNet-50 has the same 16 blocks as ResNet-34, with about 1.5 times as many layers and maps of 256 to 2,048 channels instead of 64 to 512, for about 5% more multiply-adds in the paper’s count (3.8 against 3.6 billion).

Shortcuts Between Stages

Inside a stage the shortcut is the identity. Where a block changes the resolution and the width, the input must be reshaped before the addition. He et al. compared zero-padded identity shortcuts (option A), $1 \times 1$ projections only where dimensions change (B) and projections everywhere (C), found small differences, and used option B for ResNet-50, -101 and -152; the numbers are in the residual connection article. The projection is a $1 \times 1$ convolution with stride 2, followed in TorchVision by batch normalization.

Configurations

Table 1 of He et al. (2016) defines five configurations. The name counts the weight layers on the main path: the stem convolution, two or three per block, and the fully connected layer, so ResNet-50 has $1 + 3 \cdot 16 + 1 = 50$; projection shortcuts are not counted.

Model	Block	Blocks per stage	Parameters	FLOPs (paper, multiply-adds)
ResNet-18	basic	2, 2, 2, 2	11.7M	$1.8 \times 10^9$
ResNet-34	basic	3, 4, 6, 3	21.8M	$3.6 \times 10^9$
ResNet-50	bottleneck	3, 4, 6, 3	25.6M	$3.8 \times 10^9$
ResNet-101	bottleneck	3, 4, 23, 3	44.5M	$7.6 \times 10^9$
ResNet-152	bottleneck	3, 8, 36, 3	60.2M	$11.3 \times 10^9$

The paper calls its cost figures FLOPs and defines them as multiply-adds. It prints no parameter counts in Table 1; the counts above are TorchVision’s (11,689,512, 21,797,672, 25,557,032, 44,549,160 and 60,192,808 for its 1000-class models), and the first example below reproduces them exactly from the block definitions. Zagoruyko and Komodakis (arXiv v4, Table 8) list the same 25.6M, 44.5M and 60.2M. Most of the parameters sit in the last stages and most of the computation in the middle ones: in ResNet-50, conv5_x holds 58.6% of the weights but performs 19.8% of the multiply-adds, while conv4_x, with six blocks, performs 35.8%. Liu et al. (2022) describe the heavy conv4_x stage as a choice made for compatibility with detectors whose heads operate on the $14 \times 14$ map, and call the original distribution of computation across stages largely empirical.

For CIFAR-10, He et al. used a separate, smaller design with three stages of 16, 32 and 64 filters and $6n + 2$ layers (ResNet-20 to ResNet-1202); it is described in the paper article and is the base of the CIFAR variants below.

Training

The original recipe, SGD with momentum and scale augmentation, is given in the paper article. Weights follow He et al.’s ReLU initialization; TorchVision’s defaults and the zero-initialized last batch-normalization scale of each block are covered in weight initialization.

What changed most after 2015 is the recipe, not the architecture. Gross (Facebook AI Research) and Wilber (Cornell Tech), who re-implemented ResNets in Torch (2016), replaced the paper’s scale augmentation with the scale and aspect-ratio crops of GoogLeNet and report that this improved ResNet-34’s top-1 validation error by about 1.2 points; their fb.resnet.torch code became the pipeline that ResNeXt and other papers followed. Later work trained the same ResNet-50 with newer recipes:

Training of ResNet-50	Epochs	ImageNet val top-1 (%)	Evaluation
Original models (official release)	up to about 120	75.3	single $224 \times 224$ centre crop, shorter side 256
TorchVision IMAGENET1K_V1	90	76.130	single 224 crop, resize 256
He, T. et al. (2019) baseline	120	75.87	single 224 crop, resize 256
Liu et al. (2022), DeiT-style recipe	300	78.8	224, mean of 3 seeds
Wightman et al. (2021), procedure A2	300	79.8	224, crop ratio 0.95
Wightman et al. (2021), procedure A1	600	80.4	224, crop ratio 0.95
TorchVision IMAGENET1K_V2	600	80.858	single 224 crop, resize 232

The official release (He et al., GitHub README) gives 24.7% top-1 error for ResNet-50 on a single centre crop, and 22.9% with the paper’s 10-crop testing; the release’s epoch count follows from the paper’s training length of up to $60 \times 10^4$ iterations at batch 256 (our arithmetic). TorchVision’s two rows (accuracies from the version 0.29.0 metadata) are the same architecture with different weights: the second came from a recipe that Vryniotis (2021) built up from a 90-epoch baseline with longer training (600 epochs), a cosine schedule with warmup, stronger augmentation, label smoothing, mixup and CutMix, weight averaging and other steps, each measured on ResNet-50.

Bag of Tricks. He, T. et al. (2019) measured the refinements one at a time. On ResNet-50, training with larger mini-batches (1,024 with linear learning-rate scaling and warmup), zero-γ initialization, no weight decay on biases and batch-normalization parameters, and FP16 arithmetic gave 76.21% while cutting the time per epoch from 13.3 to 4.4 minutes. On ResNet-50-D (described under Variants), cosine learning-rate decay, label smoothing ($\varepsilon = 0.1$), mixup and knowledge distillation from a ResNet-152-D teacher raised accuracy step by step from 77.26% to 79.33% (Table 6, mean of four runs; Table 1 and the abstract give 79.29%). All numbers are on the validation set with one $224 \times 224$ centre crop. The gains carried over to detection: as the backbone of Faster R-CNN, the best model raised PASCAL VOC 2007 test mAP from 77.54% to 81.33%. For semantic segmentation with an FCN on ADE20K, however, mIoU peaked at 39.33% after cosine decay and fell to 38.40% with all refinements, although ImageNet accuracy kept rising.

ResNet strikes back. Wightman, Touvron and Jégou (2021) tuned a recipe for the unmodified ResNet-50 (v1.5, the PyTorch model) at test resolution 224: LAMB optimizer with a cosine schedule, a binary cross-entropy loss that treats every class mixed in by Mixup or CutMix as present, RandAugment, and, for the longer schedules, repeated augmentation and stochastic depth. They note that published ResNet-50 results ranged from 75.2% to 79.5%. Their procedures A1, A2 and A3 (600, 300 and 100 epochs) reached 80.4%, 79.8% and 78.1%. Over 100 seeds of A2, top-1 accuracy averaged 79.72% with a standard deviation of 0.10 (range 79.50–79.98%), so differences of a few tenths between single runs are within seed noise. The same A1 recipe lifted ResNet-18 to 71.5% and ResNet-152 to 82.0%, against TorchVision’s 69.8% and 78.3%. They argue that even comparing two architectures under the same training procedure can implicitly favour one of them, because a recipe tuned for one model need not be the best for another.

ResNet-RS. Bello et al. (2021) ran an additive study on ResNet-200 at $256 \times 256$: from 79.0%, training and regularization changes alone (cosine decay, longer training, weight averaging, label smoothing, stochastic depth, RandAugment, dropout and lower weight decay) reached 82.2%, and two architectural changes, squeeze-and-excitation and ResNet-D, reached 83.4%, so about three quarters of the gain came from training (validation set, mean of two runs).

Inference

At test time a ResNet is a single forward pass: an image normalized with the training-set channel statistics, the stages, global average pooling and a softmax over the 1000 logits. Batch normalization uses fixed statistics; in the official release these were computed as plain averages over a large training batch after training, not as moving averages, which the authors report as very stable. The evaluation protocol changes the reported number by several points, so every result must carry it:

• Single centre crop: resize the shorter side to 256 and take the central $224 \times 224$ crop, the protocol of most later papers and of the tables above.
• 10-crop: average the softmax scores of the four corner crops, the centre crop and their mirrors; He et al. (2016) report their main comparisons this way.
• Fully convolutional, multi-scale: average dense scores over shorter sides of 224 to 640 pixels, used for the paper’s best single-model results.
• Larger single crop: He et al. (2016b) argued that because training crops come from images resized to a shorter side between 256 and 480, a 224 crop from a 256 image is negatively biased, and tested a $320 \times 320$ crop instead; ResNeXt and later papers report both.

As a backbone, the network is cut before the pooling, and the stage outputs are passed to a detection or segmentation head.

Performance

ImageNet ILSVRC 2012 validation results, each version from the paper that introduced it. Rows from different papers used different training pipelines, so compare within a paper.

Model (paper)	Parameters	Top-1 / top-5 error (%)	Protocol
ResNet-50 (He et al., 2016)	25.6M	22.85 / 6.71	10-crop, 224
ResNet-152 (He et al., 2016)	60.2M	21.43 / 5.71	10-crop, 224
ResNet-152 (He et al., 2016)	60.2M	19.38 / 4.49	multi-scale, fully convolutional
ResNet-200, original block (He et al., 2016b)	—	21.8 / 6.0	single 320 crop
ResNet-200, pre-activation (He et al., 2016b)	64.7M	20.7 / 5.3	single 320 crop
WRN-50-2-bottleneck (Zagoruyko & Komodakis, arXiv v4)	68.9M	21.9 / 6.03	single crop
ResNeXt-50, 32×4d (Xie et al., 2017)	25.0M	22.2 / —	single 224 crop
ResNeXt-101, 64×4d (Xie et al., 2017)	—	20.4 / 5.3	single 224 crop
ResNeXt-101, 64×4d (Xie et al., 2017)	—	19.1 / 4.4	single 320 crop
ResNet-50-D (He, T. et al., 2019)	25M	22.84 / 6.48	single 224 crop

The ResNet-50 and ResNet-152 parameter counts are those of the Configurations table, the ResNet-200 and WRN counts are from Zagoruyko and Komodakis’s Table 8, and the ResNet-50-D row converts the 77.16% / 93.52% accuracy of He, T. et al. (Table 5) into errors. Within their own papers, the baselines were 23.9% top-1 error for ResNet-50 (Xie et al.) and 24.01% (Zagoruyko and Komodakis), both single crop, which is the comparison their gains refer to. In the ILSVRC 2016 classification task, ResNeXt-based entries took second place with an ensemble of 3.03% top-5 error on the test set.

Strengths

• Regular, few hyperparameters. Depth, block type and width are the only choices; Xie et al. (2017) built ResNeXt on the same two rules.
• A multi-scale feature hierarchy at strides 4, 8, 16 and 32, with channel counts that grow as resolution falls, which detection and segmentation heads use directly.
• Fully convolutional before the head, so one set of weights serves different input sizes and crops.
• Stable training at depth. With identity shortcuts and batch normalization, networks of 50 to 200 layers train with plain SGD, and pre-activation blocks extend this to 1,001 layers on CIFAR.
• A common reference. Re-implementations agree to the parameter, and pretrained weights exist in every major framework, which is why ResNet-50 remains a baseline (Wightman et al., 2021).

Limitations

• Diminishing returns from depth. ResNet-200 with the original block had higher error than ResNet-152 while reaching lower training error, which He et al. (2016b) read as overfitting; Zagoruyko and Komodakis (2016) found wide, shallower networks more accurate and faster to train than very deep thin ones.
• Information discarded at downsampling. In the original bottleneck, the first $1 \times 1$ convolution and the projection shortcut both have stride 2, so the downsampling block never reads three quarters of its input positions; v1.5 and ResNet-D fix this (second example below).
• Coarse final maps. Stride 32 is too coarse for dense prediction without a feature pyramid, dilation or a decoder.
• Dependence on batch statistics. Batch normalization degrades with few images per GPU, as in detection; see normalization layers for the alternatives.
• Accuracy that depends on the recipe. The same ResNet-50 has been reported between 75% and 81% top-1, so comparisons that use the 2015 numbers as a baseline mix architecture with training.

Historical Importance

ResNets won the ILSVRC and COCO 2015 competitions and turned residual learning into the default way to build deep networks (see the paper article for the results and awards). As image classifiers they set the baseline that later convolutional and transformer families compared against; the Vision Transformer paper, for example, compared against large ResNets.

Their larger effect was as a backbone, the feature extractor of other vision systems. He et al. (2016) replaced VGG-16 with ResNet-101 in Faster R-CNN, and the appendix of their arXiv version describes how: sharing the layers up to conv4_x across regions at stride 16 and using conv5_x as the per-region head. The feature pyramid network takes the four stage outputs at strides 4 to 32 and combines them top-down, and Mask R-CNN was reported with ResNet and ResNeXt backbones, cut at conv4_x or combined with a feature pyramid. DeepLab dilates the last stages to keep output stride 16 or 8, DETR feeds the conv5_x map to a transformer, and CLIP trained ResNet-D image encoders with attention pooling. Transfer learning from ImageNet-pretrained ResNets became the standard starting point for these systems.

ConvNeXt (Liu et al., 2022) closed a circle: starting from ResNet-50, it changed the training recipe, the stage ratio, the stem and the blocks one step at a time to compete with Swin Transformers, and suggested from the recipe step alone (76.1% to 78.8%) that a significant part of the earlier gap between convolutional networks and vision transformers may have been due to training techniques.

Variants

ResNet v1.5 (ResNet-B)

The original release placed the stride-2 of each downsampling bottleneck in its first $1 \times 1$ convolution: in the ResNet-50 definition linked from the official README, res3a_branch2a ($1 \times 1$) has stride 2 and res3a_branch2b ($3 \times 3$) stride 1. Gross and Wilber (2016) moved the stride to the $3 \times 3$ convolution. In a ResNet-50 variant with batch normalization after the addition, they report 24.2% single-crop top-1 error with the stride in the $3 \times 3$ against 24.5% with it in the $1 \times 1$, and 24.5% for the standard batch-normalization placement with the stride in the $3 \times 3$. This placement became the default in later code under several names: NVIDIA calls it ResNet-50 v1.5 and states that it is about 0.5 points more accurate and about 5% slower in images per second; TorchVision’s Bottleneck documents it in a code comment as “ResNet V1.5” (version 0.29.0 source); He, T. et al. (2019) call it ResNet-B and measured 76.66% against 76.21% at 4.1 against 3.8 GFLOPs. The extra cost comes from the first $1 \times 1$ convolution now running at the input resolution; the first example counts 4.09 against 3.86 billion multiply-adds for ResNet-50. Unless a paper says otherwise, “ResNet-50” in code after 2016 usually means v1.5: ResNeXt states that it follows fb.resnet.torch in this (see below), and Wightman et al. (2021) note that their “vanilla” ResNet-50 is v1.5.

Pre-activation ResNet

He et al. (2016b), in “Identity Mappings in Deep Residual Networks” (ECCV 2016), moved batch normalization and ReLU before each convolution so that nothing follows the addition; why this keeps the shortcut a pure identity, and their CIFAR results up to 1,001 layers, are in the residual connection article. Two details matter when building the network: the first block takes its activation right after conv1, before the split into two paths, and the last block gets an extra BN and ReLU after its addition, before the pooling. On ImageNet, with a single $320 \times 320$ crop, pre-activation changed ResNet-152 from 21.3% to 21.1% top-1 error and ResNet-200 (16 more bottleneck blocks than ResNet-152, all at $28 \times 28$) from 21.8% to 20.7%, and to 20.1% / 4.8% with scale and aspect-ratio augmentation. ResNet-200 took about three weeks to train on 8 GPUs.

Wide ResNet

Zagoruyko and Komodakis (2016) argued that very deep, thin residual networks suffer from diminishing feature reuse and are slow to train, and studied width instead. A WRN-$n$-$k$ is a CIFAR-style network of depth $n$ with three stages of pre-activation basic blocks whose widths are multiplied by $k$; they also inserted dropout between the two convolutions of a block. On CIFAR-10 and CIFAR-100, WRN-28-10 (36.5M parameters) reached 4.17% and 20.50% test error, against 4.92% and 22.71% for the 1,001-layer pre-activation ResNet (10.2M) at the same mini-batch size of 128, and WRN-40-4, about as accurate as ResNet-1001, trained 8 times faster. The arXiv revision (v4) adds ImageNet: doubling the width of the inner $3 \times 3$ layers of ResNet-50’s bottlenecks gave WRN-50-2-bottleneck with 21.9% / 6.03% single-crop error and 68.9M parameters, better than ResNet-152 in their runs (22.16% / 6.16%, 60.2M) with a third of the layers; its summary table gives 5.79% top-5 for the same model, the figure Table 8 lists for pre-activation ResNet-200. TorchVision’s wide_resnet50_2 follows this design and has 68,883,240 parameters.

ResNeXt

Xie et al. (2017), with He as last author, replaced the bottleneck’s single path by $C$ parallel paths of the same shape whose outputs are summed, $\mathbf{y} = \mathbf{x} + \sum_{i=1}^{C} \mathcal{T}_i(\mathbf{x})$, and called the number of paths cardinality. They showed three equivalent forms of the block: separate paths summed, paths concatenated before a shared $1 \times 1$ layer, and a single bottleneck whose $3 \times 3$ layer is a grouped convolution with $C$ groups, the form they implemented. ResNeXt-50 (32×4d) has 32 paths of width 4, that is a 128-channel $3 \times 3$ convolution with 32 groups in conv2_x, and matches ResNet-50’s size: 25.0M parameters and 4.2 GFLOPs against 25.5M and 4.1 in their Table 1. Their implementation downsamples in the $3 \times 3$ layer, following fb.resnet.torch, and the first example reproduces the 4.1 billion for that placement. The reformulation is non-trivial only for blocks of depth 3 or more; a two-layer block with grouped paths collapses into one wider dense block, which is why ResNeXt uses bottlenecks.

At constant complexity, raising cardinality from 1 to 32 lowered ResNet-50’s top-1 error from 23.9% to 22.2%, and the training error fell too, which the authors read as stronger representations rather than regularization. When complexity was doubled from ResNet-101 (22.0%) to about 15 billion FLOPs, going deeper (ResNet-200, 21.7%) or wider (21.3%) helped less than raising cardinality (ResNeXt-101 2×64d, 20.7%, and 64×4d, 20.4%; all single $224 \times 224$ crop). Removing the shortcuts raised the error to 26.1% for ResNeXt-50 and to 31.2% for ResNet-50. With Torch’s generic grouped convolution, ResNeXt-101 (32×4d) took 0.95 s per mini-batch against 0.70 s for ResNet-101 on 8 GPUs, an overhead the authors expected optimized kernels to reduce.

Grouped convolution itself goes back to AlexNet, which split its layers across two GPUs. Xie et al. note that libraries such as Caffe and Torch supported grouped convolutions mainly for compatibility with AlexNet and that there had been little evidence that they improve accuracy; ResNeXt supplied that evidence. They also identify the extreme case, as many groups as channels, as the channel-wise convolution in depthwise separable convolutions, the route that MobileNets and later ConvNeXt took.

ResNet-C and ResNet-D

He, T. et al. (2019) compared three tweaks on ResNet-50 with the same training (batch 1,024, FP16): ResNet-B (v1.5, above), ResNet-C (the three-convolution stem, above) and their own ResNet-D. They observed that the stride-2 $1 \times 1$ projection of the shortcut also ignores three quarters of its input, and ResNet-D replaces it by a $2 \times 2$ average pool with stride 2 followed by a $1 \times 1$ convolution with stride 1. Applied cumulatively, each on top of the previous one, with 25M parameters in all four models, they raised top-1 accuracy from 76.21% (ResNet-50) to 76.66% (B), 76.87% (C) and 77.16% (D), at 3.8, 4.1, 4.3 and 4.3 GFLOPs, and ResNet-50-D trained only about 3% slower. CLIP’s ResNet image encoders and ResNet-RS both use ResNet-D.

SE-ResNet and ResNet-RS

Squeeze-and-excitation (Hu, Shen & Sun, 2018) adds to each residual block a small gate that rescales the channels by weights computed from their global averages. In their re-implementation, SE-ResNet-50 had 23.29% / 6.62% single-crop error against ResNet-50’s 24.80% / 7.48%, at 3.87 against 3.86 GFLOPs. ResNet-RS (Bello et al., 2021) combines ResNet-D, squeeze-and-excitation and the improved training above with scaling rules for depth, width and resolution, and reports ResNet-RS models 1.7 to 2.7 times faster on TPUs than EfficientNets of similar ImageNet accuracy.

Implementations

TorchVision provides resnet18 to resnet152, wide_resnet50_2, wide_resnet101_2 and ResNeXt models, all with the v1.5 stride placement, option B shortcuts with batch normalization, zero_init_residual (off by default) and replace_stride_with_dilation (version 0.29.0); Ross Wightman’s timm library implements many more variants, including ResNet-D, and distributes the weights trained by Wightman et al. (2021). The two examples below use only torch (PyTorch 2.14.1) and no pretrained weights.

Counting the configurations. The first example builds the five configurations from the block definitions above, counts parameters, and counts multiply-adds for a $224 \times 224$ input with forward hooks on every convolution and the fully connected layer, for both stride placements of the bottleneck. It then passes a synthetic image, a bright disk on a dark background, through ResNet-50 and prints each stage’s output shape and stride and its share of parameters and multiply-adds.

import torch
from torch import nn

torch.manual_seed(0)
torch.set_num_threads(4)

class Basic(nn.Module):
    """Two 3x3 conv-BN layers; ReLU after the addition (original post-activation block)."""
    expansion = 1
    def __init__(self, c_in, width, stride=1, stride_in_3x3=True):
        super().__init__()
        c_out = width * self.expansion
        self.f = nn.Sequential(
            nn.Conv2d(c_in, width, 3, stride, 1, bias=False), nn.BatchNorm2d(width), nn.ReLU(),
            nn.Conv2d(width, c_out, 3, 1, 1, bias=False), nn.BatchNorm2d(c_out))
        self.shortcut = nn.Identity() if stride == 1 and c_in == c_out else nn.Sequential(
            nn.Conv2d(c_in, c_out, 1, stride, bias=False), nn.BatchNorm2d(c_out))   # option B projection
    def forward(self, x):
        return torch.relu(self.f(x) + self.shortcut(x))

class Bottleneck(nn.Module):
    """1x1 reduce, 3x3, 1x1 expand (x4). v1 puts the stride in the first 1x1, v1.5 in the 3x3."""
    expansion = 4
    def __init__(self, c_in, width, stride=1, stride_in_3x3=True):
        super().__init__()
        c_out = width * self.expansion
        s1, s3 = (1, stride) if stride_in_3x3 else (stride, 1)
        self.f = nn.Sequential(
            nn.Conv2d(c_in, width, 1, s1, bias=False), nn.BatchNorm2d(width), nn.ReLU(),
            nn.Conv2d(width, width, 3, s3, 1, bias=False), nn.BatchNorm2d(width), nn.ReLU(),
            nn.Conv2d(width, c_out, 1, bias=False), nn.BatchNorm2d(c_out))
        self.shortcut = nn.Identity() if stride == 1 and c_in == c_out else nn.Sequential(
            nn.Conv2d(c_in, c_out, 1, stride, bias=False), nn.BatchNorm2d(c_out))
    def forward(self, x):
        return torch.relu(self.f(x) + self.shortcut(x))

class ResNet(nn.Module):
    def __init__(self, block, blocks_per_stage, num_classes=1000, stride_in_3x3=True):
        super().__init__()
        self.stem = nn.Sequential(nn.Conv2d(3, 64, 7, 2, 3, bias=False), nn.BatchNorm2d(64), nn.ReLU(),
                                  nn.MaxPool2d(3, 2, 1))                       # stride 4 after the stem
        stages, c_in = [], 64
        for i, (width, n) in enumerate(zip((64, 128, 256, 512), blocks_per_stage)):
            layers = []
            for j in range(n):                                                # stages 3-5 halve the map in block 1
                layers.append(block(c_in, width, 2 if (j == 0 and i > 0) else 1, stride_in_3x3))
                c_in = width * block.expansion
            stages.append(nn.Sequential(*layers))
        self.stages = nn.ModuleList(stages)
        self.head = nn.Sequential(nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(c_in, num_classes))
    def forward(self, x, return_stages=False):
        x, outs = self.stem(x), []
        for stage in self.stages:
            x = stage(x); outs.append(x)
        return outs if return_stages else self.head(x)

CONFIGS = {18: (Basic, (2, 2, 2, 2)), 34: (Basic, (3, 4, 6, 3)), 50: (Bottleneck, (3, 4, 6, 3)),
           101: (Bottleneck, (3, 4, 23, 3)), 152: (Bottleneck, (3, 8, 36, 3))}

def part_of(name):                          # "stages.2.0.f.3" -> "conv4_x", "stem.0" -> "stem"
    top = name.split(".")[0]
    return f"conv{int(name.split('.')[1]) + 2}_x" if top == "stages" else top

def mult_adds(net, size=224):
    """Multiply-adds of all convolutions and the fully connected layer for one image,
    counted with forward hooks and returned per part of the network."""
    counts = {}
    def hook(name):
        def f(m, inp, out):
            per_output = (m.kernel_size[0] * m.kernel_size[1] * m.in_channels // m.groups
                          if isinstance(m, nn.Conv2d) else m.in_features)
            counts[part_of(name)] = counts.get(part_of(name), 0) + out.numel() * per_output
        return f
    hs = [m.register_forward_hook(hook(n)) for n, m in net.named_modules() if isinstance(m, (nn.Conv2d, nn.Linear))]
    with torch.no_grad():
        net.eval()(torch.zeros(1, 3, size, size))
    for h in hs:
        h.remove()
    return counts

total = lambda d: sum(d.values())
paper_flops = {18: 1.8, 34: 3.6, 50: 3.8, 101: 7.6, 152: 11.3}          # He et al. (2016), Table 1
torchvision_params = {18: 11689512, 34: 21797672, 50: 25557032, 101: 44549160, 152: 60192808}
for depth, (block, n) in CONFIGS.items():
    net = ResNet(block, n)
    params = sum(p.numel() for p in net.parameters())
    v15 = total(mult_adds(net)) / 1e9
    line = (f"ResNet-{depth:<3d} blocks {n}: {params:>10,} parameters "
            f"(torchvision {torchvision_params[depth]:,}), {v15:5.2f} G multiply-adds")
    if block is Bottleneck:
        line += f" with stride in 3x3, {total(mult_adds(ResNet(block, n, stride_in_3x3=False))) / 1e9:5.2f} G in 1x1"
    print(line + f"; paper {paper_flops[depth]}")

# ResNet-50 (stride in the 3x3) on a synthetic 224x224 image, a bright disk: stage outputs and shares.
yy, xx = torch.meshgrid(torch.arange(224.0), torch.arange(224.0), indexing="ij")
image = ((yy - 112) ** 2 + (xx - 112) ** 2 < 60 ** 2).float().expand(1, 3, 224, 224)
net = ResNet(*CONFIGS[50]).eval()
outs = dict(zip(["conv2_x", "conv3_x", "conv4_x", "conv5_x"], net(image, return_stages=True)))
macs = mult_adds(net)
params = {"stem": net.stem, "head": net.head, **{f"conv{k}_x": st for k, st in enumerate(net.stages, start=2)}}
params = {k: sum(p.numel() for p in v.parameters()) for k, v in params.items()}
for part in ["stem", "conv2_x", "conv3_x", "conv4_x", "conv5_x", "head"]:
    shape = f"{str(tuple(outs[part].shape)):18s} stride {224 // outs[part].shape[-1]:2d}" if part in outs else " " * 28
    print(f"{part:8s} {shape}  parameters {100 * params[part] / total(params):4.1f}%,"
          f" multiply-adds {100 * macs[part] / total(macs):4.1f}%")

Output:

ResNet-18  blocks (2, 2, 2, 2): 11,689,512 parameters (torchvision 11,689,512),  1.81 G multiply-adds; paper 1.8
ResNet-34  blocks (3, 4, 6, 3): 21,797,672 parameters (torchvision 21,797,672),  3.66 G multiply-adds; paper 3.6
ResNet-50  blocks (3, 4, 6, 3): 25,557,032 parameters (torchvision 25,557,032),  4.09 G multiply-adds with stride in 3x3,  3.86 G in 1x1; paper 3.8
ResNet-101 blocks (3, 4, 23, 3): 44,549,160 parameters (torchvision 44,549,160),  7.80 G multiply-adds with stride in 3x3,  7.57 G in 1x1; paper 7.6
ResNet-152 blocks (3, 8, 36, 3): 60,192,808 parameters (torchvision 60,192,808), 11.51 G multiply-adds with stride in 3x3, 11.28 G in 1x1; paper 11.3
stem                                   parameters  0.0%, multiply-adds  2.9%
conv2_x  (1, 256, 56, 56)   stride  4  parameters  0.8%, multiply-adds 16.3%
conv3_x  (1, 512, 28, 28)   stride  8  parameters  4.8%, multiply-adds 25.1%
conv4_x  (1, 1024, 14, 14)  stride 16  parameters 27.8%, multiply-adds 35.8%
conv5_x  (1, 2048, 7, 7)    stride 32  parameters 58.6%, multiply-adds 19.8%
head                                   parameters  8.0%, multiply-adds  0.1%

With the stride in the first $1 \times 1$ convolution, as in the original release, the bottleneck counts (3.86, 7.57 and 11.28 billion) agree with the paper’s 3.8, 7.6 and 11.3 at its one-decimal precision, and are close to the 3.86, 7.58 and 11.30 GFLOPs that Hu et al. (2018) list for their re-implementations; the v1.5 count of 4.09 billion for ResNet-50 matches ResNeXt’s 4.1 for its ResNet-50 and Bag of Tricks’ 4.1 for ResNet-B. For ResNet-34 the count of 3.66 billion includes three option B projections (about 0.02 billion), without which it is 3.64 billion, the paper’s 3.6.

What a downsampling block sees. The second example builds the first block of a bottleneck stage in three versions, the original (stride in the first $1 \times 1$), v1.5 (stride in the $3 \times 3$) and ResNet-D (average pool before the projection), on an $8 \times 8$ map with 8 channels; batch normalization is omitted because it does not change which positions are connected. It marks which input positions have a nonzero gradient, separately for the main path and the shortcut, using positive weights and a positive input so that no ReLU is inactive and the map shows the wiring alone. It then feeds the block two synthetic stage inputs, vertical stripes one pixel wide on the even or on the odd columns, and measures how far each output moves from the output for a blank map.

import torch
from torch import nn

torch.manual_seed(0)
torch.set_num_threads(4)
C, W, S = 8, 4, 8           # input channels, bottleneck width, map size (S x S)

def downsampling_block(variant):
    """First block of a bottleneck stage (stride 2), BN omitted: returns (main path, shortcut)."""
    s1, s3 = (2, 1) if variant == "v1" else (1, 2)                   # v1.5 and D: stride in the 3x3
    main = nn.Sequential(nn.Conv2d(C, W, 1, s1), nn.ReLU(), nn.Conv2d(W, W, 3, s3, 1), nn.ReLU(),
                         nn.Conv2d(W, 4 * W, 1))
    if variant == "D":                                               # ResNet-D: 2x2 average pool, then 1x1
        shortcut = nn.Sequential(nn.AvgPool2d(2, 2), nn.Conv2d(C, 4 * W, 1))
    else:                                                            # 1x1 projection with stride 2
        shortcut = nn.Conv2d(C, 4 * W, 1, 2)
    return main, shortcut

def seen(path, x):
    """Input positions with a nonzero gradient. Weights and input are positive, so every ReLU is
    active and the map shows the wiring alone."""
    x = x.clone().requires_grad_(True)
    path(x).sum().backward()
    return (x.grad.abs().sum(dim=(0, 1)) > 0)

# Synthetic stage input: vertical stripes one pixel wide, on the odd or the even columns.
cols = torch.arange(S)
stripes = {p: (cols % 2 == p).float().expand(1, C, S, S).clone() for p in (0, 1)}
blank = torch.zeros(1, C, S, S)

for variant in ("v1", "v1.5", "D"):
    main, shortcut = downsampling_block(variant)
    with torch.no_grad():
        for m in (main, shortcut):
            for p in m.parameters():
                p.copy_(p.abs() + 0.01)
    x = torch.rand(1, C, S, S) + 0.1
    m_seen, s_seen = seen(main, x), seen(shortcut, x)
    block = lambda z: main(z) + shortcut(z)
    with torch.no_grad():
        gap = {p: (block(stripes[p]) - block(blank)).abs().max().item() for p in (0, 1)}
    print(f"{variant:4s}: main path sees {int(m_seen.sum()):2d}/{S * S}, shortcut {int(s_seen.sum()):2d}/{S * S} "
          f"input positions; output change from blank: even-column stripes {gap[0]:.2f}, "
          f"odd-column stripes {gap[1]:.2f}" + (" (identical outputs)" if gap[1] == 0 else ""))
    if variant == "v1":
        print("      positions the v1 block sees (#):")
        for row in (m_seen | s_seen).int().tolist():
            print("      " + " ".join("#" if v else "." for v in row))

Output:

v1  : main path sees 16/64, shortcut 16/64 input positions; output change from blank: even-column stripes 9.17, odd-column stripes 0.00 (identical outputs)
      positions the v1 block sees (#):
      # . # . # . # .
      . . . . . . . .
      # . # . # . # .
      . . . . . . . .
      # . # . # . # .
      . . . . . . . .
      # . # . # . # .
      . . . . . . . .
v1.5: main path sees 64/64, shortcut 16/64 input positions; output change from blank: even-column stripes 4.81, odd-column stripes 6.30
D   : main path sees 64/64, shortcut 64/64 input positions; output change from blank: even-column stripes 3.27, odd-column stripes 5.61

In the original block, both paths read only the 16 positions with even row and column indices, the “three-quarters” that He, T. et al. (2019) describe as ignored. Stripes on the odd columns lie entirely in the unread positions, so the block’s output is exactly the output for a blank map: no choice of weights can let this block tell the two apart. Moving the stride into the $3 \times 3$ convolution, whose $3 \times 3$ window at stride 2 covers every position, lets the main path see all 64 positions while the shortcut still sees 16; the average pool of ResNet-D makes the shortcut see all of them too. The stem is not affected, since its $7 \times 7$ convolution and $3 \times 3$ pool overlap at stride 2. In a trained network the block’s input is a feature map rather than raw stripes, and the network can learn to spread information across neighbouring positions before downsampling; the example shows only what the wiring permits. The measured accuracy effect of each change on ImageNet is 0.3 to 0.5 points in the reports of Gross and Wilber, NVIDIA and He, T. et al. cited above.