DenseNet

DenseNet, the Dense Convolutional Network, is a family of convolutional neural networks for image classification introduced by Gao Huang, Zhuang Liu, Laurens van der Maaten and Kilian Q. Weinberger at CVPR 2017. Huang and Liu contributed equally. Inside a dense block, every layer receives the feature maps of all earlier layers of the block, joined by concatenation along the channel axis, and passes its own maps to every later layer. An $L$-layer block therefore has $L(L+1)/2$ direct connections instead of $L$. ResNet adds a layer’s input to its output. DenseNet never sums: it keeps every earlier map unchanged and lets each layer add only a few new ones, set by the growth rate $k$. The paper calls the bottleneck version DenseNet-B, the compressed version DenseNet-C and the combination DenseNet-BC. The four ImageNet models, DenseNet-121, -169, -201 and -161, are all DenseNet-BC. The paper was one of two CVPR 2017 Best Paper Awards, listed on the conference’s awards page.

Problem Addressed

By 2016, convolutional networks of more than 100 layers could be trained. The authors framed the remaining problem as one of information flow: as the input or the gradient passes through many layers, it can vanish and “wash out” (§1), the problem described in vanishing gradient problem. The approaches that worked shared one trait, short paths from early to late layers:

• ResNet’s identity shortcuts and the gated bypasses of Highway Networks;
• stochastic depth (Huang, Sun, Liu, Sedra & Weinberger, 2016), which drops residual blocks at random during training;
• FractalNet, with parallel paths of different depth.

Two observations from this work shaped DenseNet. First, stochastic depth showed that many layers of a deep ResNet can be dropped without harm, which the authors read as redundancy; they say their paper was partly inspired by this observation (§2). Second, the authors argued that summing the identity with a layer’s output “may impede the information flow” (§3), and that a network passing a state from layer to layer must both change the state and carry forward what has to be preserved. ResNets make that preservation explicit by addition, and every layer has its own weights. DenseNet separates the two: what each layer adds is new feature maps, and what is preserved is every earlier map, unchanged and reachable by every later layer (§1). The authors expected this feature reuse to need fewer parameters, because no layer has to relearn maps that earlier layers already computed. The residual connection article covers the additive alternative.

Architecture

The network is a stem, a sequence of dense blocks joined by transition layers, and a classifier: global average pooling and a fully connected layer. Concatenation needs equal spatial sizes, so all downsampling happens in the transition layers between blocks (§3).

Dense Connectivity

Write $\mathbf{x}_\ell$ for the output of layer $\ell$ in a block and $H_\ell$ for its transformation. A plain network computes $\mathbf{x}_\ell = H_\ell(\mathbf{x}_{\ell-1})$, a ResNet $\mathbf{x}_\ell = H_\ell(\mathbf{x}_{\ell-1}) + \mathbf{x}_{\ell-1}$ (Eq. 1), and a DenseNet

$$\mathbf{x}_\ell = H_\ell\big([\mathbf{x}_0, \mathbf{x}_1, \ldots, \mathbf{x}_{\ell-1}]\big),$$

where $[\cdot]$ is concatenation along the channel axis (Eq. 2). Following the pre-activation ResNet of He et al. (2016b), $H_\ell$ is batch normalization, ReLU and a $3 \times 3$ convolution, in that order. The paper’s Figure 1 draws a five-layer block with $k = 4$.

Growth Rate

If every $H_\ell$ produces $k$ feature maps and the block’s input has $k_0$, layer $\ell$ receives

$$k_0 + k(\ell - 1)$$

input maps, and a block of $N$ layers outputs $k_0 + kN$. The authors call $k$ the growth rate and use small values such as $k = 12$; they describe the maps as a global state of the network to which each layer adds $k$ of its own (§3). The width of a layer’s input grows linearly along the block. In a block without a bottleneck, so do the cost of each $3 \times 3$ convolution and its parameter count. The parameters of the block therefore grow with the square of its depth.

Bottleneck Layers (DenseNet-B)

Each layer produces only $k$ maps but may read hundreds. Following Inception and ResNet, the authors put a $1 \times 1$ convolution in front of the $3 \times 3$ one to reduce its input, giving $H_\ell$ = BN-ReLU-Conv($1 \times 1$)-BN-ReLU-Conv($3 \times 3$). The $1 \times 1$ convolution produces $4k$ maps (§3). The $3 \times 3$ convolution then always reads $4k$ maps, and only the $1 \times 1$ convolution’s input keeps growing.

Transition Layers and Compression (DenseNet-C, DenseNet-BC)

A transition layer is batch normalization, a $1 \times 1$ convolution and $2 \times 2$ average pooling. (The official code and TorchVision also put a ReLU after the batch normalization.) With compression, a transition that receives $m$ maps outputs $\lfloor \theta m \rfloor$, with $0 < \theta \le 1$. The paper sets $\theta = 0.5$ and calls networks with $\theta < 1$ DenseNet-C, and with both bottleneck and compression DenseNet-BC (§3).

Configurations

CIFAR and SVHN. These networks have three dense blocks with the same number of layers, at $32 \times 32$, $16 \times 16$ and $8 \times 8$, and a stem convolution with 16 output channels, or $2k$ for DenseNet-BC (§3). The official code derives the number of layers per block from the depth $L$ as $N = (L-4)/3$, halved when the layers have bottlenecks, since each such layer has two convolutions. The paper evaluates:

• DenseNet with $(L, k) = (40, 12), (100, 12), (100, 24)$;
• DenseNet-BC with $(100, 12), (250, 24), (190, 40)$.

ImageNet. These networks are DenseNet-BC with four blocks on $224 \times 224$ inputs (Table 1):

• the stem is a $7 \times 7$ convolution with $2k$ output channels and stride 2, followed by $3 \times 3$ max pooling with stride 2;
• the blocks run at $56 \times 56$, $28 \times 28$, $14 \times 14$ and $7 \times 7$;
• after the last block come global average pooling and a 1000-way fully connected layer.

Model	$k$	Layers per block	Parameters	Multiply-adds ($224 \times 224$)
DenseNet-121	32	6, 12, 24, 16	7,978,856	$2.83 \times 10^9$
DenseNet-169	32	6, 12, 32, 32	14,149,480	$3.36 \times 10^9$
DenseNet-201	32	6, 12, 48, 32	20,013,928	$4.29 \times 10^9$
DenseNet-161	48	6, 12, 36, 24	28,681,000	$7.73 \times 10^9$

The paper prints no parameter or FLOP counts for these models. It plots them in Figure 3 and mentions “a DenseNet-201 with 20M parameters” (§4.4). The counts in the table come from the first example below and equal the metadata of TorchVision 0.29.0. The names count weight layers in our reading: two per dense layer, plus the stem, the three transition convolutions and the classifier. For DenseNet-121 that gives $2 \cdot (6 + 12 + 24 + 16) + 1 + 3 + 1 = 121$. In DenseNet-121 the blocks end with 256, 512, 1024 and 1024 maps, and the transitions halve the first three to 128, 256 and 512.

Training

All networks were trained from scratch with stochastic gradient descent, using a softmax classifier, Nesterov momentum 0.9 without dampening, weight decay $10^{-4}$ and the initialization of He et al. (2015), described in the PReLU paper article (§4.2):

• CIFAR: batch size 64 for 300 epochs. The learning rate starts at 0.1 and is divided by 10 at 50% and 75% of training. The “+” settings (C10+, C100+) use mirroring and shifting augmentation. 5,000 training images were held out for validation, and the final runs used all 50,000.
• SVHN: batch size 64 for 40 epochs, with the same learning-rate schedule, all training data, no augmentation, and a 6,000-image validation split for model selection.
• No augmentation: for C10, C100 and SVHN, a dropout layer with rate 0.2 follows every convolution except the first.
• ImageNet: 90 epochs at batch size 256. The learning rate starts at 0.1 and is divided by 10 at epochs 30 and 60. The training code was the publicly available Torch implementation for ResNet (fb.resnet.torch), with every setting left as for ResNet. DenseNet-161 did not fit in GPU memory at batch 256, so it was trained at batch 128 for 100 epochs, with a third division at epoch 90.

The authors stress that the ImageNet hyperparameters were tuned for ResNets, not DenseNets, and expected that more tuning could improve DenseNet (§4.4, §6). A footnote adds that their implementation’s memory inefficiencies prevented experiments with more than 30M parameters. Each test error was measured once for each task and model setting (§4.2).

Inference

The input is a $224 \times 224$ image on ImageNet or a $32 \times 32$ image on CIFAR and SVHN, and the output is a class distribution. On ImageNet the paper reports single-crop and 10-crop testing (§4.1). A DenseNet has the backbone shape of other ImageNet CNNs, with feature maps at strides 4 to 32, but its last map is the concatenation of a whole block. Every layer reads all earlier maps of its block at inference as well, which matters for speed (see Limitations).

As of November 2026, TorchVision 0.29.0 provides densenet121, densenet161, densenet169 and densenet201 with ImageNet weights that its metadata says were ported from LuaTorch. The listed top-1 accuracies are 74.434%, 77.138%, 75.600% and 76.896%.

Performance

CIFAR and SVHN (Table 2, test error in %):

Model	Depth	Parameters	C10	C10+	C100	C100+	SVHN
DenseNet ($k=12$)	40	1.0M	7.00	5.24	27.55	24.42	1.79
DenseNet ($k=12$)	100	7.0M	5.77	4.10	23.79	20.20	1.67
DenseNet ($k=24$)	100	27.2M	5.83	3.74	23.42	19.25	1.59
DenseNet-BC ($k=12$)	100	0.8M	5.92	4.51	24.15	22.27	1.76
DenseNet-BC ($k=24$)	250	15.3M	5.19	3.62	19.64	17.60	1.74
DenseNet-BC ($k=40$)	190	25.6M	–	3.46	–	17.18	–
Pre-activation ResNet	1001	10.2M	10.56	4.62	33.47	22.71	–
Wide ResNet	28	36.5M	–	4.17	–	20.50	–

The pre-activation ResNet’s errors without augmentation (C10 and C100) are marked in the table as run by the DenseNet authors.

The comparisons the authors drew from the table are these (§4.3):

• The 190-layer DenseNet-BC had the lowest error on C10+ and C100+.
• On C10 and C100 without augmentation, DenseNet-BC with $L = 250$ lowered the error of FractalNet with dropout and drop-path, the best earlier entry in the table, by about 30% in relative terms: from 7.33% to 5.19% and from 28.20% to 19.64%.
• The 0.8M-parameter DenseNet-BC came close to the 1001-layer pre-activation ResNet with 90% fewer parameters (4.51% against 4.62% on C10+).
• Without bottleneck or compression, error fell as depth and growth rate grew. The one sign of potential overfitting was on C10, where raising $k$ from 12 to 24 took the error from 5.77% to 5.83%.
• On SVHN, the 250-layer DenseNet-BC did not improve on the 100-layer one, which the authors suggest may be explained by overfitting on a relatively easy task.

Figure 4 compares parameter efficiency on C10+ and shows that DenseNet-BC needs about a third of the parameters of a pre-activation ResNet for comparable accuracy. The CIFAR article describes the dataset and its protocol.

ImageNet (Table 3, ILSVRC 2012 validation set, error in %, single-crop with 10-crop in parentheses):

Model	Top-1	Top-5
DenseNet-121	25.02 (23.61)	7.71 (6.66)
DenseNet-169	23.80 (22.08)	6.85 (5.92)
DenseNet-201	22.58 (21.46)	6.34 (5.54)
DenseNet-161	22.33 (20.85)	6.15 (5.30)

In Figure 3, the DenseNets reach a given single-crop top-1 error with fewer parameters and fewer FLOPs than the ResNets trained with the same code. In the authors’ examples, DenseNet-201 with 20M parameters has about the validation error of ResNet-101 with more than 40M, and a DenseNet with ResNet-50’s computation performs on par with ResNet-101 (§4.4). The README of the official repository lists single-crop top-1 errors of 25.0, 23.6, 22.5 and 22.2% for its released Torch models of the same four networks. Three of these are lower than Table 3’s 23.80, 22.58 and 22.33. The README lists them under the heading “Models in the original paper” and does not explain the difference. The ImageNet article covers the benchmark.

Feature reuse. To test whether a trained network uses its dense connections, the authors trained a DenseNet with $L = 40$, $k = 12$ on C10+. For each pair of layers, they plotted the average absolute weight with which the later layer reads the earlier one’s maps (Figure 5, §5). Their reading:

• layers spread their weights over many inputs, including the earliest ones in the block;
• transition layers read the whole preceding block;
• layers in the second and third blocks give the least weight to the transition’s outputs, which suggests those outputs are redundant, the motivation for compression;
• the classifier concentrates somewhat on the last maps of the final block.

Strengths

• Parameter efficiency. At similar error, DenseNets used fewer parameters than the ResNet variants the authors compared with, on CIFAR (Figure 4) and on ImageNet (Figure 3).
• Short gradient paths. Every layer is directly connected to the block’s output, so the loss reaches each layer through at most two or three transition layers. The authors call this implicit deep supervision (§5) and relate it to deeply supervised nets.
• Regularization on small data. The gains were largest without augmentation, and the authors observed less overfitting (§4.3).
• Reuse for dense prediction and detection from scratch. The maps of every layer stay available, which later work used for segmentation (FC-DenseNet) and for detectors trained without ImageNet pretraining (DSOD; see Variants).

Limitations

• Memory in the straightforward implementation. If each layer concatenates its inputs into a new tensor, a block of $N$ layers stores $N$ concatenations of growing width, so the memory grows with $N^2$. The official repository calls DenseNet “memory hungry” in practice and attributes this to the implementation, not the architecture. The second example below measures the effect.
• Memory access and parallelism. Lee et al. (2019) argued that DenseNet’s layers have unbalanced input and output widths, which raises memory access cost per operation. They also argued that the bottleneck splits each layer into two small sequential convolutions, which suits GPU parallelism poorly. In their Pascal VOC 2007 test comparison (trained on VOC 2007+2012 trainval), DSOD300 with a DenseNet-67 backbone ran at 35 frames per second and 73.6 mAP. Their VoVNet-27-slim backbone ran at 71 frames per second with 74.8 mAP (74.87% in their text), at 5.6 against 5.3 GFLOPs.
• Speed not proportional to FLOPs. Lee et al. also found that detectors with a DenseNet backbone, despite fewer FLOPs and parameters, spent more energy and time than those with a ResNet backbone.
• Hyperparameters inherited from ResNet on ImageNet, which the authors themselves flag (§4.4).

Historical Importance

DenseNet followed ResNet by about a year and offered the other way of combining earlier features: concatenation instead of summation. It received a Best Paper Award at CVPR 2017. Later work took up its ideas in several directions:

• efficient variants that prune or restructure the dense connections (CondenseNet, VoVNet, CSPNet);
• dense blocks inside segmentation networks (FC-DenseNet) and detectors (DSOD);
• dense connections inside other networks: PWC-Net’s flow estimator concatenates each layer’s output with its input, a design that the PWC-Net article describes with the ablation of its DenseNet connections.

YOLOv4, by Bochkovskiy with CSPNet’s first two authors, Wang and Liao, lists a CSPDarknet53 backbone, as the YOLO article records.

DenseNets, DenseNet-121 most of all, were used in several well-known medical image classifiers. As the chest X-ray analysis and Computer Vision in Radiology articles describe:

• CheXNet is a 121-layer DenseNet trained on ChestX-ray14 (Rajpurkar et al., 2017);
• CheXpert’s authors compared ResNet-152, DenseNet-121, Inception-v4 and SE-ResNeXt-101 and chose DenseNet-121 (Irvin et al., 2019);
• a study of underdiagnosis in patient subgroups trained DenseNet-121 classifiers on MIMIC-CXR, CheXpert and ChestX-ray14;
• MURA’s baseline uses a 169-layer DenseNet, as the medical image classification article describes.

Variants

DenseNet-B, DenseNet-C and DenseNet-BC

These are the paper’s own variants, described above. The official code builds DenseNet-BC by default, and TorchVision implements only DenseNet-BC.

Wide-DenseNet-BC and the larger ImageNet models

The official README recommends wide, shallow DenseNets to save memory and time. Its measurements were taken on one TITAN X with batch size 64 and no memory optimization:

Model	Parameters	C10+ error	C100+ error	Memory
DenseNet-BC ($L = 100$, $k = 12$)	0.8M	4.51%	22.27%	5,452 MB
Wide-DenseNet-BC ($L = 40$, $k = 36$)	1.5M	4.58%	22.30%	4,008 MB
Wide-DenseNet-BC ($L = 40$, $k = 48$)	2.7M	3.99%	20.29%	5,245 MB

With the memory-efficient implementation, the authors trained larger ImageNet models. The README lists top-1 errors of 22.1% for DenseNet-264 ($k = 32$), 21.2% for DenseNet-232 ($k = 48$) and 20.4% for a “DenseNet-cosine-264” with $k = 48$, all from the technical report, which could not be read for this article.

CondenseNet

Huang, Liu, van der Maaten and Weinberger (2018), with Shichen Liu as joint first author, argued that dense connectivity introduces redundancy when early features are not needed in later layers. CondenseNet makes three changes:

• learned group convolutions: during training the $1 \times 1$ convolutions are split into groups, and each group learns to drop its least important inputs; the result runs as standard group convolutions at test time;
• exponentially increasing growth rate: $k = 2^{m-1} k_0$ in block $m$;
• full dense connectivity across blocks: with downsampling where resolutions differ.

The authors report that CondenseNets use about a tenth of a DenseNet’s computation at comparable accuracy. On ImageNet, a CondenseNet of 275 million multiply-adds reached 29% top-1 error, comparable to a MobileNet that needs twice the computation.

FC-DenseNet (The One Hundred Layers Tiramisu)

Jégou et al. (2017) built a segmentation network with a downsampling path, an upsampling path and skip connections between them, as in U-Net, from dense blocks, with transition-down and transition-up modules. Concatenating every map along an upsampling path would make the full-resolution layers impractically wide. They therefore upsample only the maps produced by the preceding dense block, not the block’s input, and bring the higher-resolution information in through skip connections from the downsampling path.

On CamVid (the 367/101/233-frame split of SegNet), FC-DenseNet103 with 9.4M parameters reached 66.9% mean IoU without pretraining or post-processing. Dilation8 reached 65.3% with 140.8M parameters and ImageNet pretraining. See semantic segmentation for the task.

DSOD

Shen et al. (2017), with Zhuang Liu as joint first author, built an SSD-style single-shot detector on a DenseNet backbone and trained it from scratch. They report that the deep supervision provided by dense layer-wise connections was one of the keys to making this work, and that DSOD outperforms SSD with about half its parameters.

VoVNet

Lee et al. (2019) replaced dense aggregation with one-shot aggregation: each layer feeds only the next one, and all intermediate maps are concatenated once at the end of the module. This keeps the input width of the layers constant. Their motivation was both the efficiency argument above and a re-reading of DenseNet’s Figure 5: they hypothesized that dense connections between intermediate layers produce correlated, redundant features. VoVNet-99 backbones appear in camera-based 3D detection results on nuScenes, as the bird’s-eye-view perception article reports.

CSPNet

Wang et al. (2020) argued that in a dense block the same gradient information is reused to update many layers. Their Cross Stage Partial DenseNet splits the block’s input channels into two parts, sends one part through the dense block and joins the two through a partial transition layer. On ImageNet, in their Table 3, CSPDenseNet-201 reached 77.7% top-1 with 23.07M parameters and 8.47 BFLOPs, against 77.4%, 20.01M and 8.63 BFLOPs listed there for DenseNet-201.

Implementations

The official implementation is in Lua Torch (BSD-3-Clause licence, built on fb.resnet.torch). Its README lists implementations in Caffe, PyTorch, MXNet, TensorFlow, Keras and Chainer, among them memory-efficient versions by the authors’ group in Caffe and PyTorch. TorchVision 0.29.0 provides DenseNet-BC with an option memory_efficient.

The memory-efficient implementation. Pleiss et al. (2017) describe it in a technical report on arXiv, which could not be read for this article. What follows is from the official code’s documentation, which explains the problem through the original Torch layer:

• each layer outputs the concatenation of its input and its new maps;
• the first layer’s output is therefore held in $N$ copies, the second in $N-1$, and so on, a quadratic total.

The customized layer stores the concatenated outputs, and the gradients, in shared buffers. Its strongest mode (-optMemory 4) also shares the outputs of the batch normalization before each $1 \times 1$ convolution and recomputes them in the backward pass. That makes memory linear in depth at the cost of extra time. On a DenseNet-BC with $L = 100$, $k = 12$ on CIFAR-10 on a TitanX, the README’s table goes from 5,453 MB and 0.153 s per mini-batch for the original implementation to 1,655 MB and 0.175 s. With this mode, the 190-layer DenseNet-BC trains on a single TitanX, using 8.3 GB of 12, instead of four GPUs.

The Lua layer saves the batch-normalization running statistics before the second forward pass. A second pass in training mode would otherwise update them twice.

The PyTorch version (gpleiss/efficient_densenet_pytorch) uses checkpointing: the concatenation and batch-normalization maps are discarded in the forward pass and recomputed in the backward pass. Its README reports 15–20% extra training time and gives, for a 100-layer DenseNet-BC at batch size 64 on a Pascal Titan-X, 2.863 GB and 0.165 s per mini-batch for the naive version against 1.605 GB and 0.207 s. TorchVision’s memory_efficient option does the same with torch.utils.checkpoint.

Examples

The examples below build DenseNets in PyTorch 2.14.1 (torch only, four threads, float32, random weights, no pretrained models). The script is one file, shown in five consecutive blocks; each block’s output follows it. The whole script takes about 20–40 seconds on a laptop CPU, depending on other load.

The code follows Section 3 and Table 1 of the paper and TorchVision’s layer structure:

• a dense layer is BN-ReLU-Conv($1 \times 1$, $4k$)-BN-ReLU-Conv($3 \times 3$, $k$), with in-place ReLU and convolutions without bias;
• a transition is BN-ReLU-Conv($1 \times 1$)-average pool;
• the head is BN-ReLU-global average pool-linear.

For CIFAR it follows the official Lua code: a single $3 \times 3$ stem convolution and $N = (L-4)/3$ layers per block, halved with bottlenecks. The memory-efficient layer wraps concatenation, batch normalization, ReLU and the $1 \times 1$ convolution in torch.utils.checkpoint, as TorchVision does. The forward passes use a synthetic image, a bright disk with vertical stripes, at $224 \times 224$ or $32 \times 32$. The counts do not depend on the image.

For checking, the script also computes closed-form counts from the rules above and ends by printing PASS or FAIL for each comparison.

Counting the ImageNet configurations. The first block defines the network and counts parameters and multiply-adds. Multiply-adds are counted with forward hooks on every convolution and the linear layer, the method of the ResNet article’s first example; batch normalization, ReLU and pooling are not counted. The block compares both counts with TorchVision 0.29.0’s published metadata.

import torch
from torch import nn
import torch.utils.checkpoint as cp

torch.manual_seed(0)
torch.set_num_threads(4)

class DenseLayer(nn.Module):
    """BN-ReLU-Conv1x1(4k)-BN-ReLU-Conv3x3(k) (DenseNet-B), or BN-ReLU-Conv3x3(k) without the bottleneck.
    memory_efficient recomputes concatenation, BN, ReLU and the 1x1 conv in the backward pass."""
    def __init__(self, c_in, k, bottleneck=True, memory_efficient=False):
        super().__init__()
        self.efficient = memory_efficient and bottleneck
        if bottleneck:
            self.reduce = nn.Sequential(nn.BatchNorm2d(c_in), nn.ReLU(inplace=True), nn.Conv2d(c_in, 4 * k, 1, bias=False))
            self.conv = nn.Sequential(nn.BatchNorm2d(4 * k), nn.ReLU(inplace=True), nn.Conv2d(4 * k, k, 3, padding=1, bias=False))
        else:
            self.reduce = nn.Identity()
            self.conv = nn.Sequential(nn.BatchNorm2d(c_in), nn.ReLU(inplace=True), nn.Conv2d(c_in, k, 3, padding=1, bias=False))
    def concat_reduce(self, *features):
        return self.reduce(torch.cat(features, 1))
    def forward(self, features):                       # features: list of all earlier outputs of the block
        if self.efficient and any(f.requires_grad for f in features):
            return self.conv(cp.checkpoint(self.concat_reduce, *features, use_reentrant=False))
        return self.conv(self.concat_reduce(*features))

class DenseBlock(nn.Module):
    def __init__(self, n, c_in, k, **kw):
        super().__init__()
        self.layers = nn.ModuleList([DenseLayer(c_in + i * k, k, **kw) for i in range(n)])
    def forward(self, x):
        features = [x]
        for layer in self.layers:
            features.append(layer(features))
        return torch.cat(features, 1)

def transition(c_in, c_out):
    return nn.Sequential(nn.BatchNorm2d(c_in), nn.ReLU(inplace=True), nn.Conv2d(c_in, c_out, 1, bias=False), nn.AvgPool2d(2))

class DenseNet(nn.Module):
    def __init__(self, k, blocks, bottleneck=True, theta=0.5, stem="imagenet", k0=None, classes=1000,
                 memory_efficient=False):
        super().__init__()
        c = 2 * k if k0 is None else k0
        if stem == "imagenet":                         # 7x7/2 conv, BN, ReLU, 3x3/2 max pool (Table 1)
            layers = [nn.Conv2d(3, c, 7, 2, 3, bias=False), nn.BatchNorm2d(c), nn.ReLU(inplace=True), nn.MaxPool2d(3, 2, 1)]
        else:                                          # CIFAR: one 3x3 convolution
            layers = [nn.Conv2d(3, c, 3, 1, 1, bias=False)]
        for i, n in enumerate(blocks):
            layers.append(DenseBlock(n, c, k, bottleneck=bottleneck, memory_efficient=memory_efficient))
            c += n * k
            if i < len(blocks) - 1:
                c_out = int(theta * c)                 # compression: floor(theta * m) feature maps
                layers.append(transition(c, c_out))
                c = c_out
        layers += [nn.BatchNorm2d(c), nn.ReLU(inplace=True), nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(c, classes)]
        self.net = nn.Sequential(*layers)
        for m in self.modules():
            if isinstance(m, nn.Conv2d):
                nn.init.kaiming_normal_(m.weight)
    def forward(self, x):
        return self.net(x)

def synthetic_image(size):
    """A bright disk on a dark background with vertical stripes in the green channel."""
    yy, xx = torch.meshgrid(torch.arange(size, dtype=torch.float32), torch.arange(size, dtype=torch.float32), indexing="ij")
    disk = (((yy - size / 2) ** 2 + (xx - size / 2) ** 2) < (size / 4) ** 2).float()
    stripes = (torch.div(xx, max(size // 28, 1), rounding_mode="floor") % 2)
    return torch.stack([disk, 0.5 * stripes, 0.3 * disk + 0.2])[None]

def n_params(net):
    return sum(p.numel() for p in net.parameters())

def mult_adds(net, x):
    """Multiply-adds of every convolution and the linear layer, counted with forward hooks."""
    total = [0]
    def hook(m, inp, out):
        per_output = (m.kernel_size[0] * m.kernel_size[1] * m.in_channels // m.groups
                      if isinstance(m, nn.Conv2d) else m.in_features)
        total[0] += out.numel() * per_output
    hs = [m.register_forward_hook(hook) for m in net.modules() if isinstance(m, (nn.Conv2d, nn.Linear))]
    with torch.no_grad():
        net.eval()(x)
    for h in hs:
        h.remove()
    return total[0]

# ---------------------------------------------------------------- closed forms (for the pilot's checks)
def closed_form_imagenet(k, blocks, B=True, theta=0.5, classes=1000, size=224):
    k0 = 2 * k
    P, s = 3 * 49 * k0 + 2 * k0, size // 2
    M = s * s * 3 * 49 * k0
    s //= 2
    c, trans = k0, []
    for i, n in enumerate(blocks):
        for _ in range(n):
            if B:
                P += 2 * c + c * 4 * k + 8 * k + 36 * k * k
                M += s * s * (c * 4 * k + 36 * k * k)
            else:
                P += 2 * c + 9 * c * k
                M += s * s * 9 * c * k
            c += k
        if i < len(blocks) - 1:
            co = int(theta * c)
            trans.append((c, co))
            P += 2 * c + c * co
            M += s * s * c * co
            c, s = co, s // 2
    return P + 2 * c + c * classes + classes, M + c * classes, trans

def closed_form_cifar(L, k, B, theta, k0, classes):
    N = (L - 4) // 3 // (2 if B else 1)
    P, c = 27 * k0, k0
    for i in range(3):
        for _ in range(N):
            P += (2 * c + c * 4 * k + 8 * k + 36 * k * k) if B else (2 * c + 9 * c * k)
            c += k
        if i < 2:
            co = int(theta * c)
            P += 2 * c + c * co
            c = co
    return P + 2 * c + c * classes + classes

checks = {}
image = synthetic_image(224)
print("torch", torch.__version__)
print("synthetic image", tuple(image.shape), "min", image.min().item(), "max", image.max().item(),
      "disk pixels", int(image[0, 0].sum().item()))

# ---------------------------------------------------------------- M1, M2: ImageNet configurations (Table 1)
print("\nImageNet DenseNet-BC (Table 1), 224x224")
CONFIGS = {121: (32, (6, 12, 24, 16)), 169: (32, (6, 12, 32, 32)), 201: (32, (6, 12, 48, 32)), 161: (48, (6, 12, 36, 24))}
TV = {121: (7978856, "2.834"), 169: (14149480, "3.36"), 201: (20013928, "4.291"), 161: (28681000, "7.728")}  # TorchVision v0.29.0 metadata
for depth, (k, blocks) in CONFIGS.items():
    torch.manual_seed(0)
    net = DenseNet(k, blocks)
    p, m = n_params(net), mult_adds(net, image)
    tv_p, tv_ops = TV[depth]
    decimals = len(tv_ops.split(".")[1])
    cf_p, cf_m, _ = closed_form_imagenet(k, blocks)
    checks[f"M1 params {depth}"] = p == tv_p == cf_p
    checks[f"M2 MACs {depth}"] = (m == cf_m) and f"{m / 1e9:.{decimals}f}" == tv_ops
    print(f"DenseNet-{depth} (k={k}, blocks {blocks}): {p:>10,} parameters (TorchVision {tv_p:,}), "
          f"{m / 1e9:.3f} G multiply-adds (TorchVision {tv_ops})")

Output:

torch 2.14.1
synthetic image (1, 3, 224, 224) min 0.0 max 1.0 disk pixels 9841

ImageNet DenseNet-BC (Table 1), 224x224
DenseNet-121 (k=32, blocks (6, 12, 24, 16)):  7,978,856 parameters (TorchVision 7,978,856), 2.834 G multiply-adds (TorchVision 2.834)
DenseNet-169 (k=32, blocks (6, 12, 32, 32)): 14,149,480 parameters (TorchVision 14,149,480), 3.360 G multiply-adds (TorchVision 3.36)
DenseNet-201 (k=32, blocks (6, 12, 48, 32)): 20,013,928 parameters (TorchVision 20,013,928), 4.291 G multiply-adds (TorchVision 4.291)
DenseNet-161 (k=48, blocks (6, 12, 36, 24)): 28,681,000 parameters (TorchVision 28,681,000), 7.728 G multiply-adds (TorchVision 7.728)

The parameter counts agree exactly with TorchVision’s metadata, and the multiply-adds agree to its printed precision, so its _ops values are multiply-adds, not twice that. The paper’s “20M parameters” for DenseNet-201 is 20.01 million. DenseNet-121 and -169 need fewer multiply-adds than ResNet-50 (4.09 billion in the ResNet article’s count of TorchVision’s v1.5 model); DenseNet-201 and -161 need more. The parameter savings are larger than the computation savings: DenseNet-121 has about a third of ResNet-50’s 25.6M parameters.

Bottleneck, compression and the growth of channels. The second block builds DenseNet-121’s blocks in all four variants and reads, with forward pre-hooks, the number of maps each convolution in a dense block actually receives.

# ---------------------------------------------------------------- M3, M4: bottleneck and compression, channel growth
print("\nDenseNet-121 blocks with and without bottleneck (B) and compression (C)")
for name, B, theta in (("DenseNet", False, 1.0), ("DenseNet-B", True, 1.0), ("DenseNet-C", False, 0.5), ("DenseNet-BC", True, 0.5)):
    torch.manual_seed(0)
    net = DenseNet(32, (6, 12, 24, 16), bottleneck=B, theta=theta)
    seen = []                                           # (kernel size, input maps) of every convolution in the blocks
    hooks = [m.register_forward_pre_hook(lambda m, i: seen.append((m.kernel_size[0], i[0].shape[1])))
             for blk in net.modules() if isinstance(blk, DenseBlock) for m in blk.modules() if isinstance(m, nn.Conv2d)]
    trans_seen = []
    hooks += [t[2].register_forward_hook(lambda m, i, o: trans_seen.append((i[0].shape[1], o.shape[1])))
              for t in net.net if isinstance(t, nn.Sequential)]
    m_ = mult_adds(net, image)
    for h in hooks:
        h.remove()
    p = n_params(net)
    cf_p, cf_m, cf_trans = closed_form_imagenet(32, (6, 12, 24, 16), B=B, theta=theta)
    # expected inputs: layer l of a block with k0 input maps reads k0 + 32 (l - 1) maps
    expected_first, c = [], 64
    for i, n in enumerate((6, 12, 24, 16)):
        expected_first += [c + 32 * l for l in range(n)]
        c += 32 * n
        if i < 3:
            c = int(theta * c)
    read_first = [ch for ks, ch in seen if ks == 1] if B else [ch for ks, ch in seen]
    read_3x3 = [ch for ks, ch in seen if ks == 3]
    checks[f"M3 {name}"] = (p == cf_p) and (m_ == cf_m) and (trans_seen == cf_trans)
    checks[f"M4 {name}"] = (read_first == expected_first) and (read_3x3 == ([128] * 58 if B else expected_first))
    print(f"{name:12s} {p:>10,} parameters, {m_ / 1e9:.3f} G multiply-adds; transitions (in -> out) "
          + ", ".join(f"{a}->{b}" for a, b in trans_seen) + f"; 3x3 inputs {min(read_3x3)}..{max(read_3x3)}")
    if name == "DenseNet-BC":
        print("  DenseNet-121 block 3, input maps of layers 1..24:", read_first[18:42])

Output:

DenseNet-121 blocks with and without bottleneck (B) and compression (C)
DenseNet     20,812,904 parameters, 4.724 G multiply-adds; transitions (in -> out) 256->256, 640->640, 1408->1408; 3x3 inputs 64..1888
DenseNet-B   13,908,584 parameters, 3.917 G multiply-adds; transitions (in -> out) 256->256, 640->640, 1408->1408; 3x3 inputs 128..128
DenseNet-C   10,869,096 parameters, 3.047 G multiply-adds; transitions (in -> out) 256->128, 512->256, 1024->512; 3x3 inputs 64..992
DenseNet-BC   7,978,856 parameters, 2.834 G multiply-adds; transitions (in -> out) 256->128, 512->256, 1024->512; 3x3 inputs 128..128
  DenseNet-121 block 3, input maps of layers 1..24: [256, 288, 320, 352, 384, 416, 448, 480, 512, 544, 576, 608, 640, 672, 704, 736, 768, 800, 832, 864, 896, 928, 960, 992]

The pre-hooks confirm the growth rule: layer $\ell$ of block 3 reads $256 + 32(\ell - 1)$ maps, from 256 to 992. Without a bottleneck, the $3 \times 3$ convolutions read the whole growing input, up to 1,888 maps at the end of block 4. With a bottleneck they always read $4k = 128$ maps. Compression halves the transitions’ outputs, so without it the later blocks start from 256, 640 and 1,408 maps instead of 128, 256 and 512.

The two changes act on different parts of the cost. The bottleneck alone removes about a third of the parameters (20.8M to 13.9M) and 17% of the multiply-adds. Compression alone does more for both (10.9M, 3.05 G). Together they give DenseNet-121’s 8.0M and 2.83 G, 38% of the plain network’s parameters and 60% of its multiply-adds.

The CIFAR models of Table 2. Table 2 gives one parameter count per model for both CIFAR-10 and CIFAR-100. The plain DenseNet’s stem has 16 channels in the paper’s text but $2k$ in the official Lua code at the pinned commit. The third block therefore counts each model with 10 and 100 classes, and plain DenseNets with both stems.

# ---------------------------------------------------------------- M5: CIFAR configurations (Table 2)
print("\nCIFAR configurations (Table 2 parameters), three dense blocks on 32x32 inputs")
TABLE2 = [("DenseNet", 40, 12, 1.0), ("DenseNet", 100, 12, 7.0), ("DenseNet", 100, 24, 27.2),
          ("DenseNet-BC", 100, 12, 0.8), ("DenseNet-BC", 250, 24, 15.3), ("DenseNet-BC", 190, 40, 25.6)]
small = synthetic_image(32)
for name, L, k, paper in TABLE2:
    bc = name == "DenseNet-BC"
    N = (L - 4) // 3 // (2 if bc else 1)
    stems = [2 * k] if bc else [16, 2 * k]             # paper: 16 (2k for BC); official Lua code: 2k
    row = []
    for k0 in stems:
        for classes in (10, 100):
            torch.manual_seed(0)
            net = DenseNet(k, (N, N, N), bottleneck=bc, theta=0.5 if bc else 1.0, stem="cifar", k0=k0, classes=classes)
            with torch.no_grad():
                assert net.eval()(small).shape == (1, classes)
            p = n_params(net)
            checks[f"M5 closed form {name} L={L} k={k} k0={k0} C{classes}"] = p == closed_form_cifar(L, k, bc, 0.5 if bc else 1.0, k0, classes)
            if k0 == (2 * k if bc else 16) and classes == 10:
                checks[f"M5 Table 2 {name} L={L} k={k}"] = round(p / 1e6, 1) == paper
            row.append(f"stem {k0:2d}, {classes:3d} classes {p / 1e6:6.2f}M")
    print(f"{name:11s} L={L:3d} k={k:2d} ({N:2d} layers/block), paper {paper:4.1f}M: " + "; ".join(row))

Output:

CIFAR configurations (Table 2 parameters), three dense blocks on 32x32 inputs
DenseNet    L= 40 k=12 (12 layers/block), paper  1.0M: stem 16,  10 classes   1.02M; stem 16, 100 classes   1.06M; stem 24,  10 classes   1.06M; stem 24, 100 classes   1.10M
DenseNet    L=100 k=12 (32 layers/block), paper  7.0M: stem 16,  10 classes   6.98M; stem 16, 100 classes   7.08M; stem 24,  10 classes   7.08M; stem 24, 100 classes   7.19M
DenseNet    L=100 k=24 (32 layers/block), paper 27.2M: stem 16,  10 classes  27.25M; stem 16, 100 classes  27.46M; stem 48,  10 classes  28.07M; stem 48, 100 classes  28.28M
DenseNet-BC L=100 k=12 (16 layers/block), paper  0.8M: stem 24,  10 classes   0.77M; stem 24, 100 classes   0.80M
DenseNet-BC L=250 k=24 (41 layers/block), paper 15.3M: stem 48,  10 classes  15.32M; stem 48, 100 classes  15.48M
DenseNet-BC L=190 k=40 (31 layers/block), paper 25.6M: stem 80,  10 classes  25.62M; stem 80, 100 classes  25.82M

With 10 classes and the paper’s stem, all six models round to Table 2’s printed values (1.0, 7.0, 27.2, 0.8, 15.3 and 25.6M).

The official code’s $2k$ stem gives 1.06, 7.08 and 28.07M for the three plain DenseNets, which do not round to the table. The difference is not only the stem’s own weights: a wider stem widens every layer’s input in the first block, and through the transitions, with $\theta = 1$, every later block.

With 100 classes, five of the six models no longer round to the table’s values (1.06, 7.08, 27.46, 15.48 and 25.82M); only DenseNet-BC with $L = 100$ still rounds to 0.8M. In our reading, the table reports the CIFAR-10 counts for both datasets.

Memory of the concatenations. The fourth block measures the feature maps that autograd keeps for the backward pass of one dense block in training mode, batch 1. It counts them with torch.autograd.graph.saved_tensors_hooks: unique storages of saved four-dimensional tensors, excluding parameters and the batch-normalization statistic vectors. It compares the standard layer with the memory-efficient one.

In the standard layer, autograd keeps per layer:

• the concatenation, which batch normalization saves;
• the batch-normalization output, which the in-place ReLU and the $1 \times 1$ convolution save;
• the two $4k$-channel maps of the bottleneck.

Per pixel, a block of $N$ layers with $k_0$ input maps keeps

$$\sum_{\ell=1}^{N} 2\big(k_0 + k(\ell-1)\big) + 8kN = 2Nk_0 + kN(N-1) + 8kN$$

values. In the memory-efficient layer, the concatenation and its batch normalization are recomputed. Autograd keeps only the inputs of the checkpointed function, the block’s input and the new maps of all layers but the last, plus the bottleneck maps:

$$k_0 + k(N-1) + 8kN.$$

Both formulas are our derivation from the layer’s structure, and the code checks them against the measurement. The last two columns of the output come from the formulas, not from the measurement. They compare storing every concatenation separately, $\sum_\ell \big(k_0 + k(\ell - 1)\big)$, with one shared buffer that holds the final $k_0 + kN$ maps, the arrangement of the official Lua layer’s shared buffers.

# ---------------------------------------------------------------- M6: memory kept for backward
def saved_elements(module, x):
    """Elements of the feature maps autograd keeps for the backward pass: unique storages of saved
    4-D tensors that are not parameters (batch-norm statistics and weights excluded)."""
    params = {p.untyped_storage().data_ptr() for p in module.parameters()}
    kept = {}
    def pack(t):
        ptr = t.untyped_storage().data_ptr()
        if t.dim() == 4 and ptr not in params:
            kept[ptr] = t.untyped_storage().nbytes() // t.element_size()
        return t
    with torch.autograd.graph.saved_tensors_hooks(pack, lambda t: t):
        out = module.train()(x)
    out.square().mean().backward()
    return sum(kept.values())

def formula(k0, k, N, hw):
    standard = sum(2 * (k0 + k * l) for l in range(N)) + 8 * k * N   # concatenation and its BN-ReLU in every layer
    efficient = k0 + k * (N - 1) + 8 * k * N                        # only the block input and the new features
    return standard * hw, efficient * hw

print("\nFeature maps kept for backward in one dense block (batch 1, float32), measured with saved-tensor hooks")
cases = [("DenseNet-121 block 1", 64, 6, 56), ("DenseNet-121 block 2", 128, 12, 28),
         ("DenseNet-121 block 3", 256, 24, 14), ("DenseNet-121 block 4", 512, 16, 7)]
cases += [(f"k0=256, 14x14, N={N:2d}", 256, N, 14) for N in (6, 12, 48)]
MiB = 4 / 2 ** 20
for name, k0, N, s in cases:
    torch.manual_seed(0)
    x = torch.randn(1, k0, s, s, requires_grad=True)
    measured = {}
    for eff in (False, True):
        torch.manual_seed(0)
        measured[eff] = saved_elements(DenseBlock(N, k0, 32, memory_efficient=eff), x)
    f_std, f_eff = formula(k0, 32, N, s * s)
    checks[f"M6 {name}"] = (measured[False] == f_std) and (measured[True] == f_eff)
    concat_separate = sum(k0 + 32 * l for l in range(N)) * s * s
    concat_shared = (k0 + 32 * N) * s * s
    print(f"{name:21s} standard {measured[False] * MiB:6.2f} MiB (formula {f_std * MiB:6.2f}), "
          f"memory-efficient {measured[True] * MiB:5.2f} MiB (formula {f_eff * MiB:5.2f}), "
          f"ratio {measured[False] / measured[True]:4.2f}; concatenations {concat_separate * MiB:5.2f} MiB "
          f"vs shared buffer {concat_shared * MiB:4.2f}")

Output:

Feature maps kept for backward in one dense block (batch 1, float32), measured with saved-tensor hooks
DenseNet-121 block 1  standard  39.05 MiB (formula  39.05), memory-efficient 21.05 MiB (formula 21.05), ratio 1.85; concatenations 10.34 MiB vs shared buffer 3.06
DenseNet-121 block 2  standard  31.01 MiB (formula  31.01), memory-efficient 10.62 MiB (formula 10.62), ratio 2.92; concatenations 10.91 MiB vs shared buffer 1.53
DenseNet-121 block 3  standard  26.99 MiB (formula  26.99), memory-efficient  5.34 MiB (formula  5.34), ratio 5.06; concatenations 11.20 MiB vs shared buffer 0.77
DenseNet-121 block 4  standard   5.26 MiB (formula   5.26), memory-efficient  0.95 MiB (formula  0.95), ratio 5.53; concatenations  2.25 MiB vs shared buffer 0.19
k0=256, 14x14, N= 6   standard   4.16 MiB (formula   4.16), memory-efficient  1.46 MiB (formula  1.46), ratio 2.85; concatenations  1.51 MiB vs shared buffer 0.33
k0=256, 14x14, N=12   standard  10.05 MiB (formula  10.05), memory-efficient  2.75 MiB (formula  2.75), ratio 3.65; concatenations  3.88 MiB vs shared buffer 0.48
k0=256, 14x14, N=48   standard  81.54 MiB (formula  81.54), memory-efficient 10.50 MiB (formula 10.50), ratio 7.76; concatenations 36.18 MiB vs shared buffer 1.34

The measurements equal the formulas exactly in all seven blocks, for both implementations. The standard layer’s memory is quadratic in the number of layers, through the $kN(N-1)$ term, and the efficient layer’s is linear. On a $14 \times 14$ block with 256 input maps:

• from 6 to 48 layers, eight times as many, the standard layer’s memory grows nearly twentyfold (4.16 to 81.54 MiB);
• the efficient layer’s grows about sevenfold (1.46 to 10.50 MiB);
• the ratio between them rises from 2.85 to 7.76.

In DenseNet-121, the saving is largest in the long, narrow-map blocks 3 and 4 (about fivefold). Block 1 has only six layers, and its bottleneck maps, which both implementations keep, make up most of the efficient layer’s memory and nearly half of the standard layer’s. The concatenations alone (the last two columns) are a quarter to just under half of the standard layer’s total. A shared buffer would cut them by a factor of 3.4 (block 1) to 27 (the 48-layer block).

Same gradients, twice-updated statistics. The last block runs a full training step of DenseNet-121 in both implementations, from the same seed, on a batch of the synthetic image and its mirror image. It compares all parameter gradients and the running mean of the first batch normalization.

# ---------------------------------------------------------------- M7, M8: same gradients; batch-norm statistics
batch = torch.cat([image, image.flip(-1)])
grads, running = {}, {}
for eff in (False, True):
    torch.manual_seed(0)
    net = DenseNet(32, (6, 12, 24, 16), memory_efficient=eff).train()
    inp = batch.clone().requires_grad_(True)
    (net(inp).square().sum() / 1000).backward()
    grads[eff] = torch.cat([p.grad.flatten() for p in net.parameters()])
    first_bn = net.net[4].layers[0].reduce[0]
    running[eff] = first_bn.running_mean.norm().item()
rel = (grads[True] - grads[False]).abs().max().item() / grads[False].abs().max().item()
ratio = running[True] / running[False]
checks["M7 gradients"] = rel <= 1e-6
checks["M8 running mean ratio 1.9"] = abs(ratio - 1.9) < 1e-4
print(f"\nDenseNet-121, standard vs memory-efficient, batch of 2: max relative gradient difference {rel:.1e}")
print(f"first dense layer, running mean after one step: memory-efficient / standard = {ratio:.4f}")

print("\nPILOT CHECKS:")
for key, ok in checks.items():
    print(f"  {'PASS' if ok else 'FAIL'}  {key}")

Output:

DenseNet-121, standard vs memory-efficient, batch of 2: max relative gradient difference 0.0e+00
first dense layer, running mean after one step: memory-efficient / standard = 1.9000

PILOT CHECKS:
  PASS  M1 params 121
  PASS  M2 MACs 121
  PASS  M1 params 169
  PASS  M2 MACs 169
  PASS  M1 params 201
  PASS  M2 MACs 201
  PASS  M1 params 161
  PASS  M2 MACs 161
  PASS  M3 DenseNet
  PASS  M4 DenseNet
  PASS  M3 DenseNet-B
  PASS  M4 DenseNet-B
  PASS  M3 DenseNet-C
  PASS  M4 DenseNet-C
  PASS  M3 DenseNet-BC
  PASS  M4 DenseNet-BC
  PASS  M5 closed form DenseNet L=40 k=12 k0=16 C10
  PASS  M5 Table 2 DenseNet L=40 k=12
  PASS  M5 closed form DenseNet L=40 k=12 k0=16 C100
  PASS  M5 closed form DenseNet L=40 k=12 k0=24 C10
  PASS  M5 closed form DenseNet L=40 k=12 k0=24 C100
  PASS  M5 closed form DenseNet L=100 k=12 k0=16 C10
  PASS  M5 Table 2 DenseNet L=100 k=12
  PASS  M5 closed form DenseNet L=100 k=12 k0=16 C100
  PASS  M5 closed form DenseNet L=100 k=12 k0=24 C10
  PASS  M5 closed form DenseNet L=100 k=12 k0=24 C100
  PASS  M5 closed form DenseNet L=100 k=24 k0=16 C10
  PASS  M5 Table 2 DenseNet L=100 k=24
  PASS  M5 closed form DenseNet L=100 k=24 k0=16 C100
  PASS  M5 closed form DenseNet L=100 k=24 k0=48 C10
  PASS  M5 closed form DenseNet L=100 k=24 k0=48 C100
  PASS  M5 closed form DenseNet-BC L=100 k=12 k0=24 C10
  PASS  M5 Table 2 DenseNet-BC L=100 k=12
  PASS  M5 closed form DenseNet-BC L=100 k=12 k0=24 C100
  PASS  M5 closed form DenseNet-BC L=250 k=24 k0=48 C10
  PASS  M5 Table 2 DenseNet-BC L=250 k=24
  PASS  M5 closed form DenseNet-BC L=250 k=24 k0=48 C100
  PASS  M5 closed form DenseNet-BC L=190 k=40 k0=80 C10
  PASS  M5 Table 2 DenseNet-BC L=190 k=40
  PASS  M5 closed form DenseNet-BC L=190 k=40 k0=80 C100
  PASS  M6 DenseNet-121 block 1
  PASS  M6 DenseNet-121 block 2
  PASS  M6 DenseNet-121 block 3
  PASS  M6 DenseNet-121 block 4
  PASS  M6 k0=256, 14x14, N= 6
  PASS  M6 k0=256, 14x14, N=12
  PASS  M6 k0=256, 14x14, N=48
  PASS  M7 gradients
  PASS  M8 running mean ratio 1.9

The gradients are identical to the last bit. In our reading, recomputation performs the same arithmetic in the same order on this CPU, so the efficient layer changes the memory, not the result.

The running mean of the first batch normalization is not identical. It is 1.9 times larger in the efficient network: the checkpointed batch normalization runs a second time in training mode during the backward pass, so its running statistics are updated twice with momentum 0.1, $0.1\mu + 0.9 \cdot 0.1\mu = 0.19\mu$ instead of $0.1\mu$. This is why the official Lua code saves the statistics before the second forward pass and restores them. Nothing restores them in the PyTorch layer of this example. TorchVision’s code at version 0.29.0 makes the same checkpoint call and has no restore step either, so in our reading it updates them twice as well. Training is unaffected, since batch normalization uses batch statistics there. The statistics used at evaluation are averaged differently. From a running value $r$, the two updates with the same batch give $0.9(0.9r + 0.1\mu) + 0.1\mu = 0.81r + 0.19\mu$, so in our reading each training step acts as a single update with momentum 0.19 instead of 0.1, and the running statistics weight recent batches more.