Model Compression

Model compression turns a trained neural network into one that needs less memory, storage, computation or energy, while keeping as much of its accuracy as possible. Five families of methods do this. Pruning removes weights, or whole filters and channels, that contribute little. Quantization stores weights and activations in fewer bits. Knowledge distillation trains a small network to imitate a large one. Low-rank factorization replaces a layer’s weight matrix by a product of two thinner ones. Compact architectures are designed to be cheap from the start, so there is less to remove afterwards. The methods combine: Deep Compression (Han, Mao & Dally, 2016) prunes, quantizes and then entropy-codes the same network.

Bucilă, Caruana and Niculescu-Mizil (2006) used the term, as the title of their paper, for training a small neural network to mimic a large ensemble of classifiers, the idea that distillation later generalized. For a vision system the reason is usually deployment: a detector or segmenter that must run on a phone, a camera, a robot or a car, within a power budget and a frame time, or a server model that must answer many requests cheaply.

Purpose

A trained network carries more parameters than its task needs, and the cost of running it is set by the memory it must read as much as by the arithmetic it must do. Han et al. (2015) quote energy figures for a 45 nm process, from a table by Horowitz: a 32-bit floating-point add costs 0.9 pJ, a 32-bit read from on-chip SRAM 5 pJ, and a 32-bit read from off-chip DRAM 640 pJ. A model too large for on-chip memory therefore pays mostly for moving weights, and they estimate that a network of one billion connections run at 20 frames per second would need 12.8 W for DRAM access alone. Shrinking a model can bring it into cache, reduce the bandwidth it needs, shorten download and update sizes, and, where the hardware supports the result, reduce the number of operations per image.

The five families attack different costs. Unstructured pruning and entropy coding reduce storage. Structured pruning, low-rank factorization, distillation into a smaller network and compact architectures reduce the operations that dense hardware executes. Quantization reduces both bytes and the cost of each operation on hardware with fast low-precision arithmetic. Which cost matters depends on the deployment, so a compression method is chosen for a target, not in general.

When to Use It

Compression is worth its effort when a model that is accurate enough does not fit the target: too large to ship or to keep in memory, too slow for the frame budget, or too power-hungry for the battery or the thermal envelope. It helps less, or not at all, in three cases:

• The target cannot exploit the result. A model with 90% of its weights set to zero is no faster on hardware and libraries that compute dense products, and an INT8 model is no faster where no INT8 kernels exist. Li et al. (2017) moved from weight pruning to filter pruning for this reason, and Mishra et al. (2021) describe the speed gain of unstructured sparsity as negligible and at times negative.
• The network is already compact. Blalock et al. (2020), plotting results from the pruning literature on ImageNet, found that pruning improves the accuracy–efficiency trade-off of a given architecture, but generally helps less than switching to a better architecture, and helps most on architectures that are inefficient to begin with. Han et al.’s large reductions came largely from the fully connected layers of AlexNet and VGG-16, and Li et al. (2017) point out that more recent designs replace such layers with average pooling.
• The task needs precision that the method removes. Every method trades accuracy for efficiency, and the cost has to be measured on the task and data the model will see.

Blalock et al.’s finding suggests choosing a compact architecture such as MobileNet or EfficientNet first, and then compressing that architecture with pruning, quantization or distillation.

How It Works

Pruning

Pruning sets parameters to zero and keeps them there. Blalock et al. (2020) write a pruned model as $f(x; M \odot W')$, where $M \in \{0, 1\}^{|W'|}$ is a binary mask, $W'$ the remaining weights after any further training, and $\odot$ the elementwise product. Nearly all methods in their survey follow one loop: train the network, give each parameter or group a score, remove the lowest-scoring ones, fine-tune to recover the lost accuracy, and repeat. Methods differ in four choices.

Structure. Unstructured pruning removes individual weights and leaves a sparse tensor of the original shape. Structured pruning removes whole groups, such as filters, channels or neurons, and leaves a smaller dense network. Mishra et al. (2021) note that unstructured pruning tends to keep more accuracy than structured pruning of a network to the same size, but only structured pruning produces networks that hardware and libraries optimized for dense computation can run faster (Blalock et al., 2020). Between the two, N:M sparsity requires that every group of M consecutive weights contain at most N nonzeros. Mishra et al. describe the 2:4 pattern supported by the Sparse Tensor Cores of NVIDIA’s Ampere GPUs: half the weights are zero, each kept value carries a 2-bit index, and the hardware doubles the math throughput of matrix products whose first operand is in this format.

Score. The simplest score is the magnitude of a weight. Optimal Brain Damage (LeCun, Denker & Solla, 1989) argued that magnitude is only a proxy and scored each parameter by the predicted increase in the training loss when it is deleted. Approximating the loss by a second-order Taylor expansion around a trained minimum (where the gradient vanishes) and ignoring the off-diagonal terms of the Hessian $H$, the saliency of parameter $u_k$ is

$$s_k = \tfrac{1}{2} h_{kk} u_k^2,$$

with $h_{kk}$ the $k$-th diagonal element, which they computed by a backpropagation-like pass of the same cost as the gradient. On their digit-recognition network, deleting by saliency raised the training objective much less than deleting by magnitude. For filters, Li et al. (2017) ranked each filter by the sum of the absolute values of its weights, its $\ell_1$ norm, and found that pruning the smallest filters worked better than pruning random or the largest ones.

Scope and schedule. Scores can be compared within each layer, removing the same fraction from every layer, or globally across the network. Pruning can happen in one step or gradually. Han et al. (2015) found iterative pruning and retraining raised the parameter reduction of AlexNet without loss from 5× to 9× compared with a single aggressive step. Blalock et al. list as consistent findings that magnitude pruning compresses networks substantially without loss of accuracy and that many methods beat random pruning at large amounts of pruning.

Fine-tuning. Han et al. (2015) call the retraining step critical: without it accuracy drops at a third of the original connections rather than at a tenth. They also state that it is better to keep the surviving weights than to reinitialize them, reasoning from earlier findings on co-adapted features. Li et al. (2017) similarly found that a pruned architecture trained from scratch was less accurate than the pruned and retrained model (Table 1).

The lottery ticket hypothesis. Frankle and Carbin (2019) asked why the small networks found by pruning are hard to train from scratch. They pruned a trained network by magnitude, reset the surviving weights to their original initial values, and retrained: these subnetworks, which they call winning tickets, matched the original network’s test accuracy at 10–20% of its size or less on MNIST and CIFAR-10, while the same sparse structures with fresh random initial values trained worse. The hypothesis states that a dense, randomly initialized network contains such a subnetwork. Their experiments used unstructured pruning on small datasets; they note that finding tickets required training a network 15 or more times, and that larger datasets such as ImageNet were not studied.

Quantization

Quantization stores each weight or activation as a low-bit integer or float plus a scale, so INT8 divides storage and memory traffic by four relative to FP32 and runs on integer arithmetic where the hardware has it. The arithmetic, calibration, quantization-aware training and formats are the subject of the model quantization article. It applies equally to a model that has already been pruned, distilled or chosen to be compact. Deep Compression applied a different, non-uniform form after pruning: it clustered each layer’s weights with k-means and stored a small codebook of shared values plus a short index per weight, then fine-tuned the shared values (Han et al., 2016). This saves storage, but the authors note that the libraries of the time could not run the indexed format, so they did not benchmark its speed.

Knowledge distillation

Distillation trains a small student network on the outputs of a large teacher, instead of, or as well as, on the labels. Bucilă et al. (2006) labelled a large set of unlabelled or synthetic examples with an ensemble and trained a neural network on them; averaged over eight problems, their mimic networks were 1,000 times smaller and faster than the ensembles, usually with negligible loss in performance. Hinton, Vinyals and Dean (2015) observed that the small probabilities a trained classifier assigns to wrong classes carry information about how it generalizes: which 2s look like 3s and which look like 7s. To expose that information they raise the temperature $T$ of the teacher’s softmax,

$$q_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)},$$

and train the student, at the same temperature, on the cross-entropy with these soft targets; the student uses $T = 1$ after training. With labels available, they add the ordinary cross-entropy at $T = 1$ with a considerably lower weight, and multiply the soft term by $T^2$, because its gradients scale as $1/T^2$. In the limit of high temperature, with logits centred to zero mean, matching soft targets becomes matching the teacher’s logits by squared error, the objective Hinton et al. attribute to Caruana and colleagues (§2.1).

Because distillation changes only the training targets, the student’s architecture is free (our observation): it can be a compact architecture, a pruned or factorized version of the teacher, or a network that will be quantized, and the teacher can be an ensemble, as in both papers.

Low-rank factorization

A fully connected layer with weight matrix $W \in \mathbb{R}^{m \times n}$ costs $mn$ parameters and multiply-adds per input. Its truncated singular value decomposition keeps the $r$ largest singular values,

$$W \approx U_r \Sigma_r V_r^\top = \underbrace{\left(U_r \Sigma_r^{1/2}\right)}_{m \times r} \underbrace{\left(\Sigma_r^{1/2} V_r^\top\right)}_{r \times n},$$

which is the best rank-$r$ approximation in the Frobenius norm. The layer becomes two layers, first $r \times n$ and then $m \times r$, with $r(m + n)$ parameters, fewer than $mn$ when $r < mn/(m + n)$. Denton et al. (2014) bound the error of the layer’s output by the first discarded singular value times the norm of the input. A convolutional kernel, a four-dimensional tensor, is first folded into a matrix or decomposed with tensor methods. Denton et al. applied such approximations to the first two convolutional layers and the fully connected layers of an ImageNet network and fine-tuned the layers above. Jaderberg, Vedaldi and Zisserman (2014) approximated a layer’s filters by combinations of a small basis of filters that are rank-1, that is separable, in space; in their second scheme the layer becomes a layer of vertical filters followed by a layer of horizontal ones.

A depthwise separable convolution is the same idea built into the architecture: a standard convolution whose kernel is constrained to factor into a per-channel spatial filter and a $1 \times 1$ channel mixer.

Compact architectures

Rather than compressing a large network, one can train a small one designed for efficiency. MobileNet builds almost every layer from depthwise separable convolutions and adds width and resolution multipliers that trade accuracy for cost; later versions add inverted residual blocks and architecture search driven by measured latency on phones. EfficientNet scales a small searched baseline in depth, width and input resolution together. Mishra et al. (2021) group MobileNet v2, SqueezeNet, MNASNet, ShuffleNet v2 and EfficientNet B0 together as small, parameter-efficient networks. The MobileNet and EfficientNet articles give the designs and results of the two families linked here. Such networks gain less from pruning (Blalock et al., 2020) and lose more to quantization than large ones, as the quantization article reports for MobileNets and EfficientNets. Under their default 2:4 workflow, Mishra et al. found these small networks lost about 1 to 4 points of ImageNet top-1 accuracy, while all 20 classifiers of their Table 2 stayed within 0.4 point of dense; permuting the channels before fine-tuning recovered the loss (Table 3).

Combining methods

Deep Compression (Han et al., 2016) chains three steps: magnitude pruning with retraining, k-means weight sharing with fine-tuning of the shared values, and Huffman coding of the weight indices and the gaps between nonzero positions. Their Figure 1 shows pruning reducing storage by 9–13×, pruning plus quantization by 27–31× and the full pipeline by 35–49×, with the sparse-index overhead included (the caption gives 10× for pruning). They report that pruning and quantization did not interfere: in their Figure 7, accuracy began to drop at the same number of bits whether or not AlexNet had been pruned.

How to Apply It

The example trains a small convolutional network on the 8×8 digit images bundled with scikit-learn (1,797 images, a fixed stratified split of 1,257 for training and 540 for testing), and compresses it in four ways: unstructured magnitude pruning, filter pruning, a truncated SVD of the largest layer, and INT8 weights. For each it reports test accuracy, the number of nonzero parameters, multiply-accumulates (MACs) per image counted by PyTorch’s FLOP counter, the bytes of the saved state dictionary and of the same file compressed with zlib. It runs with PyTorch 2.14.1 and scikit-learn 1.9.1 in one thread with fixed seeds, and two runs printed identical output. Wall-clock time is not measured: on a model this small it would mostly reflect framework overhead rather than the method.

import copy, io, sys, warnings, zlib
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.nn.utils.prune as prune
from torch.utils.flop_counter import FlopCounterMode
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

SEED = int(sys.argv[1]) if len(sys.argv) > 1 else 0
torch.manual_seed(SEED)
torch.set_num_threads(1)
torch.use_deterministic_algorithms(True)

# ---------------------------------------------------------------- data: 8x8 digit images
X, y = load_digits(return_X_y=True)                  # 1,797 images, pixel values 0..16
X = torch.tensor(X / 16.0, dtype=torch.float32).reshape(-1, 1, 8, 8)
y = torch.tensor(y)
itr, ite = train_test_split(np.arange(len(y)), test_size=0.3, stratify=y.numpy(), random_state=0)
xtr, ytr, xte, yte = X[itr], y[itr], X[ite], y[ite]
print("torch", torch.__version__, "| seed", SEED, "| train", len(itr), "test", len(ite))

class CNN(nn.Module):
    def __init__(self, c1=16, c2=32, hidden=64):
        super().__init__()
        self.conv1 = nn.Conv2d(1, c1, 3, padding=1)
        self.conv2 = nn.Conv2d(c1, c2, 3, padding=1)
        self.fc1 = nn.Linear(c2 * 16, hidden)
        self.fc2 = nn.Linear(hidden, 10)
    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)          # 8x8 -> 4x4
        return self.fc2(F.relu(self.fc1(x.flatten(1))))

def train(model, epochs, lr=3e-3):
    opt = torch.optim.Adam(model.parameters(), lr)
    for _ in range(epochs):
        model.train()
        perm = torch.randperm(len(ytr))
        for i in range(0, len(ytr), 64):
            idx = perm[i:i + 64]
            loss = F.cross_entropy(model(xtr[idx]), ytr[idx])
            opt.zero_grad(); loss.backward(); opt.step()
    return model.eval()

def accuracy(model):
    with torch.no_grad():
        return (model(xte).argmax(1) == yte).float().mean().item() * 100

def nonzero(model):
    return sum(int((p != 0).sum()) for p in model.state_dict().values() if p.is_floating_point())

def macs(model):
    with FlopCounterMode(display=False) as fc, torch.no_grad():
        model(xte[:1])
    return fc.get_total_flops() // 2                         # PyTorch counts a multiply-add as 2 FLOPs

def file_bytes(model):
    buf = io.BytesIO()
    torch.save(model.state_dict(), buf)
    raw = buf.getvalue()
    return len(raw), len(zlib.compress(raw, 9))

def report(name, model):
    raw, gz = file_bytes(model)
    print(f"{name:28s} {accuracy(model):5.1f}%  {nonzero(model):7,d}  {macs(model):8,d}  {raw:8,d}  {gz:8,d}")

base = train(CNN(), 30)
print(f"\n{'model':28s} {'acc':>6s}  {'nonzero':>7s}  {'MACs':>8s}  {'file B':>8s}  {'zlib B':>8s}")
report("baseline CNN", base)

torch 2.14.1 | seed 0 | train 1257 test 540

model                           acc  nonzero      MACs    file B    zlib B
baseline CNN                  98.3%   38,282   337,536   156,173   143,980

The parameters and the operations sit in different layers. The first fully connected layer, fc1, holds 32,832 of the 38,282 parameters (86%) but performs 32,768 of the 337,536 MACs (10%); the second convolution holds 4,640 parameters (12%) and performs 294,912 MACs (87%), because each of its weights is applied at all 64 positions of the 8×8 map. Li et al. (2017) describe the same split in VGG-16, whose fully connected layers hold 90% of the parameters and less than 1% of the operations. The FP32 file is about 4 bytes per parameter plus some archive overhead, and zlib shrinks it by only 8%, because the low-order bytes of trained floating-point weights are close to random (our explanation).

The first experiment prunes the smallest weights globally, across all four layers, with prune.global_unstructured, which replaces each weight by a product of the original tensor and a mask buffer. The network is fine-tuned for five epochs with the masks fixed; prune.remove then makes the zeros permanent.

# ---------------------------------------------------------------- unstructured magnitude pruning
def weights(m):
    return [(m.conv1, "weight"), (m.conv2, "weight"), (m.fc1, "weight"), (m.fc2, "weight")]

warnings.filterwarnings("ignore", "Sparse CSR tensor support is in beta")

def csr_bytes(w):
    """Bytes of a weight matrix in PyTorch's CSR layout (int64 indices, float32 values)."""
    s = w.detach().reshape(w.shape[0], -1).to_sparse_csr()
    return sum(t.numel() * t.element_size() for t in (s.crow_indices(), s.col_indices(), s.values()))

print("\nGlobal magnitude pruning of all weights, then 5 epochs of fine-tuning with the mask fixed")
print(f"{'model':28s} {'acc':>6s}  {'nonzero':>7s}  {'MACs':>8s}  {'file B':>8s}  {'zlib B':>8s}"
      f"   no-FT acc   fc1 dense B / CSR B")
for amount in (0.5, 0.8, 0.9, 0.95):
    m = copy.deepcopy(base)
    prune.global_unstructured(weights(m), pruning_method=prune.L1Unstructured, amount=amount)
    acc0 = accuracy(m)
    train(m, 5, lr=1e-3)
    for mod, name in weights(m):
        prune.remove(mod, name)                       # make the zeros permanent
    w = m.fc1.weight
    print(f"{f'pruned {amount:.0%}':28s} {accuracy(m):5.1f}%  {nonzero(m):7,d}  {macs(m):8,d}  "
          f"{file_bytes(m)[0]:8,d}  {file_bytes(m)[1]:8,d}   {acc0:7.1f}%    "
          f"{w.numel() * 4:,d} / {csr_bytes(w):,d}")

Global magnitude pruning of all weights, then 5 epochs of fine-tuning with the mask fixed
model                           acc  nonzero      MACs    file B    zlib B   no-FT acc   fc1 dense B / CSR B
pruned 50%                    98.5%   19,202   337,536   156,237    85,459      98.5%    131,072 / 178,024
pruned 80%                    98.1%    7,754   337,536   156,237    43,147      96.1%    131,072 / 55,672
pruned 90%                    98.5%    3,938   337,536   156,237    27,826      78.3%    131,072 / 22,216
pruned 95%                    93.7%    2,030   337,536   156,237    19,475      41.7%    131,072 / 7,912

Three things stay the same at every sparsity. The saved file keeps its size, to within 64 bytes, because zeros are stored as 4-byte floats like any other value; the MAC count is unchanged, because the counter, like a dense kernel, multiplies by the zeros; and so the model is no faster on dense hardware. What changes is what an encoder can exploit. The zlib-compressed file shrinks with the number of nonzeros, and storing fc1 in compressed sparse row (CSR) format pays only above a certain sparsity: with PyTorch’s 8-byte indices each nonzero costs 12 bytes instead of 4, so at 50% sparsity the CSR matrix is larger than the dense one, and by our arithmetic it breaks even only when fewer than a third of the weights remain. Deep Compression therefore stores the gaps between nonzero positions rather than absolute indices, in 4 bits for AlexNet and 5 bits for VGG-16 in its Tables 4 and 5 (its Section 2 gives 8 bits for convolutional and 5 for fully connected layers), and Mishra et al. contrast the index overhead of CSR, up to 200% for 8-bit values, with the 2 bits per value of the 2:4 format. Fine-tuning does the rest: without it, 90% sparsity costs 20 points of accuracy; with five epochs, none.

The second experiment removes whole filters of conv2, ranked by their $\ell_1$ norm as in Li et al. (2017). prune.ln_structured only zeroes them, so the network is then rebuilt with fewer filters, and the matching input columns of fc1 are dropped.

# ---------------------------------------------------------------- structured (filter) pruning
print("\nFilter pruning of conv2 by L1 norm (masked, then physically removed), 5 epochs of fine-tuning")
for amount in (0.5, 0.75):
    m = copy.deepcopy(base)
    prune.ln_structured(m.conv2, "weight", amount=amount, n=1, dim=0)  # zero whole filters
    masked_acc, masked_macs = accuracy(m), macs(m)
    alive = m.conv2.weight_mask.flatten(1).any(1)                      # surviving filters
    keep, drop = alive.nonzero().flatten(), (~alive).nonzero().flatten()
    prune.remove(m.conv2, "weight")
    def fc1_cols(ch):                                                    # fc1 inputs of these maps
        return (ch[:, None] * 16 + torch.arange(16)).flatten()
    small = CNN(c2=len(keep))                                           # a genuinely smaller network
    small.conv1.load_state_dict(m.conv1.state_dict())
    small.conv2.weight.data = m.conv2.weight.data[keep].clone()
    small.conv2.bias.data = m.conv2.bias.data[keep].clone()
    small.fc1.weight.data = m.fc1.weight.data[:, fc1_cols(keep)].clone()
    # A zeroed filter still outputs relu(bias), a constant map: fold it into fc1's bias.
    const = F.relu(m.conv2.bias.data[drop]).repeat_interleave(16)
    small.fc1.bias.data = m.fc1.bias.data + m.fc1.weight.data[:, fc1_cols(drop)] @ const
    small.fc2.load_state_dict(m.fc2.state_dict())
    acc0 = accuracy(small)
    train(small, 5, lr=1e-3)
    print(f"  {amount:.0%} of filters: masked {masked_acc:.1f}% at {masked_macs:,d} MACs | "
          f"removed {acc0:.1f}% -> fine-tuned {accuracy(small):.1f}%, "
          f"{nonzero(small):,d} weights, {macs(small):,d} MACs, {file_bytes(small)[0]:,d} B")

Filter pruning of conv2 by L1 norm (masked, then physically removed), 5 epochs of fine-tuning
  50% of filters: masked 93.1% at 337,536 MACs | removed 93.1% -> fine-tuned 98.0%, 19,578 weights, 173,696 MACs, 81,357 B
  75% of filters: masked 92.0% at 337,536 MACs | removed 92.0% -> fine-tuned 97.2%, 10,226 weights, 91,776 MACs, 43,981 B

The masked network still performs every MAC; the rebuilt one performs 51% of them at 50% of the filters and 27% at 75%, and its file shrinks with it, with no sparse format needed. The rebuilt network gives exactly the masked network’s accuracy only because the bias of each removed filter is folded into the next layer: ln_structured masks the weights but not the biases, so a “pruned” filter still emits the constant $\mathrm{ReLU}(b)$, and dropping it without the fold changes the outputs. Before fine-tuning, removing half the filters (93.1%) costs more accuracy than removing half the weights individually (98.5%), at nearly the same number of remaining weights, the pattern Mishra et al. (2021) describe for structured against unstructured pruning.

The third experiment replaces fc1 by its truncated SVD, two thinner layers initialized from $U_r \Sigma_r^{1/2}$ and $\Sigma_r^{1/2} V_r^\top$, and prints the fraction of the squared singular values (“energy”) each rank keeps.

# ---------------------------------------------------------------- low-rank factorization of fc1
print("\nTruncated SVD of fc1 (64 x 512) into two thinner layers, 5 epochs of fine-tuning")
U, S, Vh = torch.linalg.svd(base.fc1.weight.detach(), full_matrices=False)
for r in (32, 16, 8, 4):
    m = copy.deepcopy(base)
    first = nn.Linear(512, r, bias=False)
    second = nn.Linear(r, 64)
    first.weight.data = (S[:r, None].sqrt() * Vh[:r]).clone()          # sqrt(S_r) V_r^T
    second.weight.data = (U[:, :r] * S[:r].sqrt()).clone()              # U_r sqrt(S_r)
    second.bias.data = base.fc1.bias.data.clone()
    m.fc1 = nn.Sequential(first, second)
    energy = (S[:r] ** 2).sum() / (S ** 2).sum()
    acc0 = accuracy(m)
    train(m, 5, lr=1e-3)
    print(f"  rank {r:2d}: {energy:.1%} of energy | {acc0:.1f}% -> fine-tuned {accuracy(m):.1f}%, "
          f"{nonzero(m):,d} weights, {macs(m):,d} MACs")

Truncated SVD of fc1 (64 x 512) into two thinner layers, 5 epochs of fine-tuning
  rank 32: 91.1% of energy | 98.1% -> fine-tuned 98.3%, 23,946 weights, 323,200 MACs
  rank 16: 83.6% of energy | 98.1% -> fine-tuned 98.3%, 14,730 weights, 313,984 MACs
  rank  8: 73.3% of energy | 91.9% -> fine-tuned 97.6%, 10,122 weights, 309,376 MACs
  rank  4: 48.4% of energy | 53.9% -> fine-tuned 93.0%, 7,818 weights, 307,072 MACs

Rank 16 keeps the accuracy with 38% of the original parameters, and even rank 4, which keeps under half the energy and starts at 53.9%, recovers to 93.0% with fine-tuning. But the MACs fall by at most 9%: fc1 held most of the parameters and few of the operations, so factorizing it is a storage method for this network. Making it faster would mean factorizing or pruning conv2, the layer that holds the operations; Denton et al. (2014) likewise factorized convolutional layers for speed and fully connected layers for memory.

Finally, for comparison, the weights are quantized to symmetric per-channel INT8, the scheme the quantization article derives, and the integers, scales and biases are saved as a device would store them.

# ---------------------------------------------------------------- INT8 weights, for comparison
print("\nSymmetric per-channel INT8 weights (activations stay FP32)")
m, packed = copy.deepcopy(base), []
for mod, name in weights(m):
    w = getattr(mod, name).data
    shape = (-1,) + (1,) * (w.dim() - 1)
    scale = w.abs().flatten(1).amax(1) / 127                            # one scale per output channel
    q = torch.clamp(torch.round(w / scale.view(shape)), -127, 127).to(torch.int8)
    packed.append((q, scale, mod.bias.data))                            # what a device would store
    w.copy_(q.float() * scale.view(shape))                              # dequantized, for evaluation
buf = io.BytesIO()
torch.save(packed, buf)
n = sum(p.numel() for p in m.parameters())
print(f"  {accuracy(m):.1f}%, {n:,d} weights, {macs(m):,d} MACs, {len(buf.getvalue()):,d} B")

Symmetric per-channel INT8 weights (activations stay FP32)
  98.1%, 38,282 weights, 337,536 MACs, 42,815 B

INT8 makes the file 3.6 times smaller with no retraining and keeps every weight, so the MAC count is unchanged; each MAC becomes cheaper only on hardware that runs integer kernels.

The script takes the seed as an argument and was run with seeds 0 to 4; the data split is the same in every run. The baseline reached 98.0% to 98.5%. After 90% unstructured pruning and fine-tuning, accuracy was 97.2% to 98.5%, at most 1.1 points below the baseline of the same run, against 69.1% to 86.1% before fine-tuning; at 95% it fell to 90.9% to 94.3%. Removing 75% of conv2’s filters left 97.2% to 98.5% after fine-tuning, rank 4 gave 93.0% to 95.2%, and INT8 weights stayed within 0.2 point of the baseline. On 540 test images one image is 0.19 point and the binomial standard error near 98% is about 0.6 point, so the differences between the milder settings and the baseline are within noise; the costs at 95% sparsity and rank 4 are not.

Comparing Compressed Models

A compressed model is only as good as the comparison that supports it. Blalock et al. (2020) aggregated results from 81 pruning papers and found the comparisons so fragmented that they could not tell which methods were best: a quarter of the papers compared with no other pruning method, the most common dataset and architecture pair, VGG-16 on ImageNet, appeared in only 22 of the 81, and of the results they plotted for the most common pairs only one came with any measure of spread. Their recommendations, and the lessons of the sources above, amount to a checklist.

• Accuracy on the same split, with the same recipe. Compare against the uncompressed model evaluated on the same test set and trained with the same recipe, data augmentation and library; Blalock et al. list the initial model, augmentation, random variation, the pruning and fine-tuning schedule and the deep-learning library as confounders. Report several operating points, they suggest compression ratios of 2, 4, 8, 16 and 32, and means with standard deviations over runs, since many methods change accuracy by less than one point.
• Parameters, and what “pruned” means. Report the number of nonzero parameters and the compression ratio. Blalock et al. define the ratio as original size divided by compressed size, and note that papers also use it for one minus the inverse, and that “Pruned%” sometimes means the fraction removed and sometimes the fraction left.
• Bytes, including overhead. Parameter counts ignore the index overhead of sparse formats and the codebooks and scales of quantized ones; Deep Compression’s ratios include both. The example shows a 50% sparse layer that takes more bytes than the dense one.
• Operations, with the counting rule. Report multiply-accumulates and say so. Blalock et al. define theoretical speedup as the original number of multiply-adds divided by the new number, found up to a factor of four between the FLOPs different papers reported for AlexNet on ImageNet (371, 724 and 1,500 MFLOPs), and showed on ResNet-18 that global pruning beat layerwise pruning at a fixed model size while the reverse held at a fixed theoretical speedup (their Figure 6). The EfficientNet article explains why “FLOPs” in some papers means multiply-adds and in others twice that.
• Memory. Peak activation memory and memory traffic, not only weight storage, decide whether a model fits and how fast and efficiently it runs, and an operation count says nothing about either (our reasoning, from the energy figures in Han et al., 2015).
• Measured latency on named hardware. Speed claims need the device, the runtime and its version, the numeric precision, the batch size and the thread count. Operation counts do not predict time: the MobileNet article shows models with equal multiply-adds and different phone latencies. Li et al. (2017) measured filter-pruned networks on a Titan X (Pascal) GPU with cuDNN 5.1 at batch size 128: VGG-16 saved 40.7% of the inference time for 34.2% fewer FLOPs, but ResNet-110 saved only 21.8% for 38.6% fewer (their Table 3), although the text calls the time savings close to the FLOP reductions. End-to-end latency of a camera system, beyond the model alone, is measured as described in real-time vision pipelines.
• Sparsity that dense kernels cannot use. Dense kernels gain nothing from unstructured zeros, and sparse kernels gain only in some regimes. In Deep Compression’s timings of AlexNet’s and VGG-16’s fully connected layers, the pruned layers ran on average 3×, 3.5× and 4.2× faster than dense ones on the CPU, desktop GPU and mobile GPU at batch size 1 (Figure 9; the text says 3× to 4×), where the computation is a matrix-vector product limited by memory, but at batch size 64 (Appendix A, Table 8) the sparse kernels were slower than the dense ones for every layer on the desktop GPU and CPU, and for four of the six layers on the mobile GPU; on the Titan X, AlexNet’s first fully connected layer took 19.8 µs per image dense and 94.6 µs sparse. Only a sparsity pattern the hardware supports, such as 2:4 on Sparse Tensor Cores, turns zeros into speed: Mishra et al. measured close to 2× on large INT8 matrix products on an A100, with smaller gains on smaller ones (their Figure 3).

Best Practices

• Start from the target. Decide whether the constraint is storage, memory, operations, latency or energy on a specific device, and choose methods that reduce that cost on that device.
• Fine-tune after every lossy step and keep the surviving weights rather than reinitializing them (Han et al., 2015; Li et al., 2017). Mishra et al. retrained 2:4-pruned networks with the original schedule and hyperparameters and matched dense accuracy on most ImageNet classifiers they tested.
• Measure sensitivity per layer. Han et al. (2015) and Li et al. (2017) pruned one layer at a time to see how much each tolerates. Han et al. gave the most sensitive layer, the first convolution, the smallest pruning threshold; Li et al. pruned sensitive layers less or not at all.
• Remove structure physically. Masks are a training device; a deployed model should be rebuilt with the smaller shapes, with the biases of removed channels folded into the next layer, as in the example, and their batch-normalization parameters removed, as Li et al. did.

Pitfalls

• Compressing the wrong layer. Parameters and operations concentrate in different layers, as the example shows; factorizing or pruning a fully connected layer saves memory, not time, in a network dominated by convolutions.
• Comparing against a weak or ambiguous baseline. LeCun, Denker and Solla (1989) stressed that their starting point was a state-of-the-art network, since a poor one is easy to improve. Blalock et al. found networks called “VGG-16” or “ResNet-50” that differed between papers.
• Ignoring the old literature. Blalock et al. note that magnitude pruning was described by Janowsky in 1989 and later reintroduced, and that most early work beyond Optimal Brain Damage and Optimal Brain Surgeon is no longer cited.

History

Blalock et al. (2020) date pruning to the late 1980s; Optimal Brain Damage (LeCun, Denker & Solla, 1989) gave it a second-order criterion. Bucilă et al. (2006) used the name model compression for transferring an ensemble into a small network. With deep convolutional networks the subject returned: low-rank approximations of trained CNNs in 2014 (Denton et al.; Jaderberg et al.), distillation with temperature and order-of-magnitude pruning of AlexNet and VGG-16 in 2015 (Hinton et al.; Han et al.), and Deep Compression in 2016. Li et al. (2017) turned to filter pruning because unstructured sparsity did not speed up dense libraries, the year MobileNet v1 appeared. Frankle and Carbin (2019) reframed pruning as finding trainable subnetworks, Blalock et al. (2020) audited the field, and NVIDIA’s Ampere GPUs added 2:4 sparsity to their matrix units (Mishra et al., 2021).

Evidence and Impact

Results reported by the sources, on the datasets and metrics they used:

• Magnitude pruning (Han et al., 2015, ImageNet). AlexNet went from 61 million to 6.7 million parameters (9×) with top-1 error 42.77% against 42.78%, and VGG-16 from 138 million to 10.3 million (13×) with top-1 error 31.34% against 31.50% (Table 1). The operations fell by 3× for AlexNet and 5× for VGG-16 (Tables 4 and 5), less than the parameters, because most removed weights were in the fully connected layers. Retraining the pruned AlexNet took 173 hours against 75 hours for the original training. The caption of their Table 5 gives VGG-16’s parameter reduction as 12×, while Table 1, the abstract and the 7.5% of weights remaining give 13×.
• Deep Compression (Han et al., 2016, ImageNet). AlexNet shrank from 240 MB to 6.9 MB (35×) with unchanged top-1 error (42.78%), and VGG-16 from 552 MB to 11.3 MB (49×) with top-1 error 31.17% against 31.50% (Table 1). AlexNet used 8-bit indices into 256 shared values for convolutional layers and 5-bit indices into 32 values for fully connected layers (Table 4).
• Filter pruning (Li et al., 2017). On CIFAR-10, VGG-16 lost 34.2% of its FLOPs and 64.0% of its parameters with error 6.60% against 6.75%, and ResNet-110 lost 38.6% of its FLOPs with error 6.70% against 6.47%; on ImageNet, ResNet-34 lost 24.2% of its FLOPs with top-1 error 27.83% against 26.77% (Table 1).
• Low-rank approximation. Denton et al. (2014) report empirical speedups of 2–2.5× on each of the first two convolutional layers of an ImageNet network, on CPU and GPU, with less than 1% drop in performance. Their Table 2 gives 5.8–13.4× fewer parameters in the three fully connected layers by SVD at the lower ranks tried, for 0.19–0.84% more error, and 2.4–3.5× at the higher ranks; the introduction gives 5–10× and the discussion 5–13×. They found speedups close to the theoretical operation savings hard to achieve in practice. Jaderberg et al. (2014) report a 2.5× speedup with no loss in accuracy and 4.5× with less than 1% drop on a scene-text character recognition network.
• Distillation (Hinton et al., 2015). On MNIST, a large regularized network made 67 test errors and a small unregularized one 146; the small network trained to match the large one’s soft targets at $T = 20$ made 74. On a speech recognition acoustic model, a distilled single model reached 60.8% frame accuracy against 61.1% for the 10-model ensemble that taught it and 58.9% for the baseline, with the same word error rate as the ensemble (10.7% against 10.9% for the baseline; Table 1).
• 2:4 sparsity (Mishra et al., 2021, ImageNet top-1). With the prune-once-and-retrain workflow, ResNet-50 reached 76.2% sparse in FP16 and INT8 against 76.1% dense (Table 2); they attribute such differences to run-to-run variation.