Model Quantization

Model quantization runs a trained neural network with its weights and activations stored in a low-precision format, most often 8-bit integers (INT8), instead of the 32-bit floating point (FP32) it was trained in. Each tensor of integers carries a real-valued scale, and often an integer zero point, that map the integers back to real numbers, so that convolutions and matrix products run as integer multiply-adds and only a few rescaling steps touch real numbers. Going from 32 to 8 bits divides the memory for weights and activations by four, and on hardware with fast integer or 8-bit arithmetic it shortens latency and saves energy. The price is a small error added to every quantized tensor, which costs accuracy unless the quantizer is chosen carefully or the network is trained to tolerate it.

The word has other meanings in computer vision. Colour quantization and vector quantization replace pixel colours or descriptors by the nearest of a few prototypes, usually found by k-means clustering, as the visual vocabulary of the bag of visual words does. Signal quantization is the rounding of a measured voltage to integer levels by the analog-to-digital converter of an image sensor, which gives a digital image its bit depth. Model quantization uses the same rounding as the converter, but applies it to a network’s parameters and intermediate results. It is one of the ways to make a network smaller and faster. Pruning removes weights and knowledge distillation trains a small network to imitate a large one; model compression covers the family as a whole.

Purpose

A trained network is a long chain of multiply-accumulate operations on FP32 numbers. Nagel et al. (2021) point out that these operations and the movement of their operands from memory consume most of the energy of inference, and that the cost of digital arithmetic grows linearly to quadratically with the number of bits. Krishnamoorthi (2018) lists what 8 bits buy: a model four times smaller, less working memory and better cache use for activations, faster arithmetic on most processors, and four times less data moved per value. Wu et al. (2020) tabulate the relative peak throughput of tensor operations on NVIDIA’s Turing GPUs as 1× for FP32, 8× for FP16, 16× for INT8 and 32× for INT4.

Quantization buys this without redesigning the network, which distinguishes it from compact architectures such as MobileNet. Jacob et al. (2018) argued that it should be judged on such architectures, which are already efficient and leave little redundancy to remove, and on real devices: they quantized MobileNets and measured them on phone processors.

When to Use It

Quantization helps when a model must run under limits on latency, memory, energy or bandwidth: on phones, in cameras, in robots and cars, or in data centres serving many requests. It pays only where the target hardware and its software run the chosen format faster. Jacob et al. found the gain to depend on the relative speed of integer and floating-point arithmetic on each processor: on the Snapdragon 821, whose floating point was better optimized, the latency gain was smaller than on the Snapdragon 835. Weight-only quantization still makes the model four times smaller where integer arithmetic is not supported (Krishnamoorthi, 2018).

In the sources summarized under Evidence and Impact, INT8 post-training quantization is close to free for most convolutional networks once weights are per-channel and activations calibrated; networks with depthwise separable layers or outlier activations need more care, and 4-bit weights or activations need quantization-aware training and still lose accuracy on compact models. Krishnamoorthi expects regression tasks such as super-resolution and HDR image processing to need more than 4 or 8 bits.

How It Works

The affine mapping

Jacob et al. (2018) define quantization as an affine map from integers $q$ to real numbers $r$,

$$r = S\,(q - Z),$$

with a positive real scale $S$ and an integer zero point $Z$, the integer that represents the real value 0. To quantize a tensor to $b$ bits, a real range $[r_{\min}, r_{\max}]$ that contains 0 is chosen, and

$$S = \frac{r_{\max} - r_{\min}}{2^b - 1}, \qquad Z = \operatorname{round}\!\left(-\frac{r_{\min}}{S}\right), \qquad q = \operatorname{clamp}\!\left(\operatorname{round}\!\left(\frac{r}{S}\right) + Z,\; 0,\; 2^b - 1\right),$$

where clamp limits a value to the integer grid; the dequantized value $\hat r = S(q - Z)$ approximates $r$ (Jacob et al., Eq. 12; Nagel et al., 2021, Eqs. 4–7). This is affine or asymmetric quantization. The range must contain 0 exactly, because zero padding must not pick up error (Jacob et al.; Krishnamoorthi), nor the zeros produced by ReLU (Nagel et al.): a range such as $(2.1, 3.5)$ is relaxed to $(0, 3.5)$ first.

Symmetric (scale-only) quantization fixes $Z = 0$ and uses a range $[-\alpha, \alpha]$ with signed integers. Wu et al. use the restricted range $[-127, 127]$ for INT8, $S = \alpha / 127$, so the grid is symmetric. Jacob et al., whose weight ranges are otherwise affine, also keep int8 weights in $[-127, 127]$; in Appendix B of the arXiv version they explain that without $-128$ every product of two int8 values is below $2^{14}$ in magnitude, so two products can be summed in a 16-bit register without overflow. Nagel et al. and Gholami et al. also describe the full range $[-128, 127]$, and an unsigned symmetric grid suits one-sided data such as ReLU outputs. Power-of-two scales, $S = 2^{-k}$, turn rescaling into bit shifts at the cost of a coarser choice of range (Nagel et al., §2.2.2).

Conventions differ between sources. Wu et al. define the scale as the reciprocal, $s = (2^b - 1)/(\alpha - \beta)$, and dequantize by dividing; Gholami et al. write $Q(r) = \operatorname{Int}(r/S) - Z$, with the opposite sign of $Z$. The arithmetic is the same.

Rounding and clipping error

Every quantized value carries a rounding error of at most $S/2$ in magnitude; values outside the range are clipped to its ends. Widening the range removes clipping error but enlarges $S$ and so the rounding error of every other value (Nagel et al., §2.2). Choosing the range, calibration, is this trade-off. Modelling the rounding error as uniform over $[-S/2, S/2]$, the same model image noise uses for the converter, gives an error variance of $S^2/12$; each bit removed doubles $S$ and so quarters that variance, about 6 dB of signal-to-quantization-noise ratio (SQNR) per bit (our arithmetic, under that model).

Integer-only arithmetic

The scheme of Jacob et al., the one adopted in TensorFlow Lite according to the paper, lets a whole layer run on integers. For a matrix product $r_3 = r_1 r_2$ of $N \times N$ matrices, substituting $r_\alpha = S_\alpha (q_\alpha - Z_\alpha)$ gives

$$q_3^{(i,k)} = Z_3 + M \sum_{j=1}^{N} \left(q_1^{(i,j)} - Z_1\right)\left(q_2^{(j,k)} - Z_2\right), \qquad M = \frac{S_1 S_2}{S_3}.$$

The only non-integer is the multiplier $M$, which depends on the scales alone and is computed offline. Jacob et al. found it always in $(0, 1)$ in practice and write it as $M = 2^{-n} M_0$ with $M_0 \in [0.5, 1)$: $M_0$ is stored as the int32 nearest to $2^{31} M_0$, which keeps at least 30 bits of relative accuracy, and the factor $2^{-n}$ is a rounding right shift. Expanding the product,

$$q_3^{(i,k)} = Z_3 + M\left(N Z_1 Z_2 - Z_1 a_2^{(k)} - Z_2 \bar{a}_1^{(i)} + \sum_{j=1}^{N} q_1^{(i,j)} q_2^{(j,k)}\right), \qquad a_2^{(k)} = \sum_j q_2^{(j,k)}, \quad \bar{a}_1^{(i)} = \sum_j q_1^{(i,j)},$$

leaves the core cost in the integer product sum, $2N^3$ operations, while the zero-point terms take $O(N^2)$. The products of 8-bit values are accumulated in int32. The bias is stored as int32 with scale $S_1 S_2$ and zero point 0, so it adds directly to the accumulator; Jacob et al. keep it at 32 bits because an error in a bias is added to many outputs and acts as a systematic offset. The accumulator is then multiplied by $M_0$, shifted, offset by $Z_3$ and saturated to $[0, 255]$, and a ReLU or ReLU6 is just a tighter clamp.

When both operands are affine, one zero-point term depends on the input and must be computed at run time; Nagel et al. compare its cost to an extra channel, which is why weights are usually quantized symmetrically (see Best Practices). A network with batch normalization folds it into the preceding convolution before the weights are quantized, so the quantized weights are those the device will use (Jacob et al., §3.2). Additions and concatenations need their inputs on a common scale; in the appendix of the arXiv version, Jacob et al. rescale the inputs of an addition and require all inputs and the output of a concatenation to share quantization parameters.

Granularity

A single scale per weight or activation tensor is per-tensor (per-layer) quantization. Per-channel quantization gives each output channel of a weight tensor, each convolution kernel, its own scale. Both keep the integer product intact: in $y_{ij} \approx S_x S_{w,j} \sum_k x^q_{ik} w^q_{kj}$ the scales can be factored out of the sum over $k$ as long as the activation scale does not vary with $k$ and the weight scale varies only with the output column $j$ (Wu et al., §3.2; Nagel et al., §2.4.2). Per-channel activation scales would vary along the summed dimension and require rescaling inside the accumulation, so activations are quantized per tensor.

Per-channel weights matter because channels can differ widely. Jacob et al. name ranges that differ more than 100× between the output channels of a layer as a common failure of simple post-training quantization, and Krishnamoorthi traces such variation to folded batch normalization. Finer still, per-group or block quantization gives a scale to every block of consecutive weights. Nagel et al. note that most fixed-point accelerators did not support it in 2021; as of October 2026, TensorRT 11.4.1 requires per-block scales for INT4 weights, with blocks of 64 or 128.

Calibration

Weight ranges come from the weights themselves. Activation ranges depend on the inputs, so post-training quantization runs a calibration set of representative images through the network and records each activation tensor. The usual criteria, as the sources describe them:

• Min-max uses the observed minimum and maximum. It never clips, but a single outlier stretches the range for every other value (Nagel et al., §3.1). Wu et al.’s “max” calibration, the largest absolute value, is the symmetric form.
• Percentile clips a fixed fraction of the largest magnitudes, for example 99.99% (Wu et al., after McKinstry et al.).
• MSE chooses the range that minimizes the squared error between a tensor and its quantized version, by grid search or golden-section search (Nagel et al., Eq. 16). For the logits of a classifier, Nagel et al. minimize the cross-entropy between the softmax of the original and the quantized logits instead, since the order of the largest values matters more than the error of the many small ones.
• Entropy (KL divergence) chooses the range that minimizes the Kullback–Leibler divergence between the distribution of the original values and that of the quantized values; Wu et al. describe it as the default calibration of TensorRT in 2020.
• Batch-normalization based ranges, $[\min(\beta - \alpha\gamma), \max(\beta + \alpha\gamma)]$ from the learned shift $\beta$ and scale $\gamma$ of the preceding batch normalization, need no data (Nagel et al., Eqs. 18–19, with $\alpha = 6$).

Krishnamoorthi found about 100 mini-batches enough for activation ranges to converge, Nagel et al. usually 500 to 1,000 unlabelled images, and Wu et al. used 1,024 training images for their ImageNet models. Static quantization fixes the activation ranges after calibration; dynamic quantization computes them from each input at run time, which is more accurate but costs computation (Gholami et al., §III-D).

Quantization-aware training

Quantization-aware training (QAT) simulates quantization during training so that the weights adapt to it. Each tensor to be quantized passes through a fake quantization (simulated quantization) operation, quantize then dequantize, so the forward pass sees the rounding and clipping of the deployed model while all arithmetic stays in floating point (Jacob et al., §3; Wu et al., Eq. 12). Weights are stored and updated in floating point, so that small gradient steps accumulate instead of being rounded away (Krishnamoorthi, §3.2).

Rounding has zero derivative almost everywhere, so backpropagation through it needs an approximation. The straight-through estimator (STE) treats the rounding as the identity in the backward pass; with clipping, the gradient of the fake-quantized value $\hat{x}$ with respect to its input $x$ becomes

$$\frac{\partial \hat{x}}{\partial x} = \begin{cases} 1, & q_{\min} \le x \le q_{\max}, \\ 0, & \text{otherwise}, \end{cases}$$

where $[q_{\min}, q_{\max}]$ is the representable real range (Nagel et al., Eq. 37; Krishnamoorthi, Eq. 15; Wu et al., Eq. 13). Bengio, Léonard and Courville (2013) named the estimator, crediting the idea of backpropagating through a hard threshold as if it were the identity to a 2012 lecture by Hinton; Krishnamoorthi cites BinaryConnect for it instead. Later methods, among them PACT and LSQ, learn the clipping range or scale along with the weights (Gholami et al., §III-G; Nagel et al., §4.1; Wu et al., §5.3).

Jacob et al. derived weight ranges from the current minimum and maximum of the weights and activation ranges from exponential moving averages, and disabled activation quantization for the first 50,000 to 2 million steps while those ranges settled. Batch normalization complicates QAT because training uses batch statistics while the deployed model uses folded long-term statistics. Krishnamoorthi corrects the folded weights to the long-term statistics and later freezes them; Nagel et al. report that simply folding batch normalization statically into a pretrained model works as well or better (their Table 7).

Number formats

Floating-point formats spend their bits on an exponent and a mantissa and cover a wide range with relative precision; integer formats have a uniform grid and need a scale. FP16 and bfloat16 (BF16) halve FP32 storage; BF16 keeps FP32’s 8 exponent bits and so its range, and Micikevicius et al. (2022) describe it as single precision with fewer bits. Both are also used to train networks (Wu et al., 2020). FP8 has two encodings, proposed by authors from NVIDIA, Arm and Intel: E4M3 for weights and activations and E5M2 for gradients. E4M3 gives up infinities and most NaN encodings to reach a maximum of 448 instead of 240 (Micikevicius et al., 2022). As with integers, FP8 tensors are multiplied by a per-tensor scale before casting, so that their largest values land near the format’s maximum. INT4 and the 4-bit float FP4 (E2M1) halve INT8 again and are used with per-block scales. As of October 2026, TensorRT 11.4.1 uses symmetric quantization only, with INT8, FP8 E4M3, INT4 for weights only, and FP4; its MXFP8 and NVFP4 schemes give every block of 32 or 16 values its own scale, computed at run time for activations.

How to Apply It

The example trains a small convolutional network from scratch on rendered 32×32 images of four shapes, folds its batch normalization, and then quantizes it, implementing the quantize and dequantize steps explicitly as fake quantization. It runs with PyTorch 2.14.1 in a single thread with fixed seeds, and two runs printed identical output.

import sys
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

SEED = int(sys.argv[1]) if len(sys.argv) > 1 else 0
torch.manual_seed(SEED)
torch.set_num_threads(1)
torch.use_deterministic_algorithms(True)
rng = np.random.default_rng(SEED)
print("torch", torch.__version__, "seed", SEED)

# ---------------------------------------------------------------- data
def render(kind, n=32):
    """One grayscale image of a disk, square, triangle or ring, at random size,
    position, rotation and contrast, with sensor-like noise."""
    yy, xx = np.mgrid[0:n, 0:n] + 0.5
    cx, cy = rng.uniform(10, 22, 2)
    r = rng.uniform(5, 9)
    th = rng.uniform(0, 2 * np.pi)
    u = (xx - cx) * np.cos(th) + (yy - cy) * np.sin(th)
    v = -(xx - cx) * np.sin(th) + (yy - cy) * np.cos(th)
    if kind == 0:
        m = u**2 + v**2 < r**2
    elif kind == 1:
        m = (abs(u) < 0.8 * r) & (abs(v) < 0.8 * r)
    elif kind == 2:
        m = (v > -0.5 * r) & (v < r - np.sqrt(3) * abs(u))
    else:
        m = (u**2 + v**2 < r**2) & (u**2 + v**2 > (0.55 * r) ** 2)
    bg, fg = rng.uniform(0.1, 0.4), rng.uniform(0.6, 0.95)
    img = np.where(m, fg, bg) + rng.normal(0, 0.05, (n, n))
    return np.clip(img, 0, 1)

def make(n):
    y = rng.integers(0, 4, n)
    x = np.stack([render(k) for k in y])[:, None]
    x = (x - 0.5) / 0.5                      # inputs in [-1, 1]
    return torch.tensor(x, dtype=torch.float32), torch.tensor(y)

xtr, ytr = make(3000)
xte, yte = make(1000)
xcal = xtr[:256]                             # calibration images

# ---------------------------------------------------------------- FP32 model
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.c1, self.b1 = nn.Conv2d(1, 16, 3, padding=1), nn.BatchNorm2d(16)
        self.c2, self.b2 = nn.Conv2d(16, 32, 3, padding=1), nn.BatchNorm2d(32)
        self.c3, self.b3 = nn.Conv2d(32, 32, 3, padding=1), nn.BatchNorm2d(32)
        self.fc = nn.Linear(32, 4)
    def forward(self, x):
        x = F.max_pool2d(F.relu(self.b1(self.c1(x))), 2)
        x = F.max_pool2d(F.relu(self.b2(self.c2(x))), 2)
        x = F.relu(self.b3(self.c3(x))).mean((2, 3))
        return self.fc(x)

net = Net()
opt = torch.optim.Adam(net.parameters(), 3e-3)
sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, 15 * 47)
for epoch in range(15):
    perm = torch.randperm(3000)
    for i in range(0, 3000, 64):
        idx = perm[i:i + 64]
        loss = F.cross_entropy(net(xtr[idx]), ytr[idx])
        opt.zero_grad(); loss.backward(); opt.step(); sched.step()
net.eval()

# Fold each BatchNorm into its convolution, as an inference engine does.
def fold(conv, bn):
    s = bn.weight / torch.sqrt(bn.running_var + bn.eps)
    return (conv.weight * s[:, None, None, None]).detach(), \
           (bn.bias + (conv.bias - bn.running_mean) * s).detach()

W = [fold(net.c1, net.b1), fold(net.c2, net.b2), fold(net.c3, net.b3),
     (net.fc.weight.detach(), net.fc.bias.detach())]

torch 2.14.1 seed 0

The quantizers follow the formulas above: symmetric INT8 or INT4 weights in the restricted range, per tensor or per output channel, and affine unsigned activations with a zero point. The forward pass quantizes the input of every convolution and of the final linear layer, which is where an integer engine would hold 8-bit data; max pooling needs no requantization because it only selects values already on the grid (Nagel et al., §2.3.3). Biases stay in floating point, standing in for their int32 form. The first check compares the hand-written weight quantizer with PyTorch’s built-in torch.fake_quantize_per_channel_affine.

# ---------------------------------------------------------------- quantizers
def fq_sym(w, bits, per_channel):
    """Symmetric (scale-only) fake quantization of weights, restricted range."""
    qmax = 2 ** (bits - 1) - 1                    # 127 for INT8, 7 for INT4
    if per_channel:
        amax = w.abs().amax(dim=tuple(range(1, w.dim())), keepdim=True)
    else:
        amax = w.abs().max()
    s = amax / qmax
    return torch.clamp(torch.round(w / s), -qmax, qmax) * s

def qparams(a, b, bits):
    """Affine (asymmetric) scale and zero point for the range [a, b]."""
    a, b = min(a, 0.0), max(b, 0.0)               # real 0 must be representable
    s = (b - a) / (2 ** bits - 1)
    z = int(np.clip(round(-a / s), 0, 2 ** bits - 1))
    return s, z

def fq_aff(x, s, z, bits):
    """Affine fake quantization: quantize to unsigned integers, dequantize."""
    q = torch.clamp(torch.round(x / s) + z, 0, 2 ** bits - 1)
    return (q - z) * s

def calibrate(t, bits, method):
    t = t.flatten()
    if method == "min-max":
        return qparams(t.min().item(), t.max().item(), bits)
    if method == "percentile":                   # clip 0.01% at each end
        lo, hi = torch.quantile(t, torch.tensor([1e-4, 1 - 1e-4])).tolist()
        return qparams(lo, hi, bits)
    # "mse": shrink the min-max range and keep the one with least squared error
    lo, hi = t.min().item(), t.max().item()
    best = None
    for k in range(20, 101):
        s, z = qparams(lo * k / 100, hi * k / 100, bits)
        err = ((fq_aff(t, s, z, bits) - t) ** 2).mean().item()
        if best is None or err < best[0]:
            best = (err, s, z)
    return best[1], best[2]

# ---------------------------------------------------------------- quantized forward
def forward(x, Wq, act=None, abits=8):
    """Folded network. Wq: list of (weight, bias); act: list of four (s, z)
    activation quantizers for the inputs of conv1, conv2, conv3 and fc."""
    taps = []
    def aq(t, i):
        taps.append(t)
        return t if act is None else fq_aff(t, *act[i], abits)
    x = aq(x, 0)
    x = F.max_pool2d(F.relu(F.conv2d(x, *Wq[0], padding=1)), 2); x = aq(x, 1)
    x = F.max_pool2d(F.relu(F.conv2d(x, *Wq[1], padding=1)), 2); x = aq(x, 2)
    x = F.relu(F.conv2d(x, *Wq[2], padding=1)).mean((2, 3));     x = aq(x, 3)
    return F.linear(x, *Wq[3]), taps

def acc(Wq, act=None, abits=8):
    with torch.no_grad():
        return (forward(xte, Wq, act, abits)[0].argmax(1) == yte).float().mean().item() * 100

def sqnr(w, wq):
    return 10 * torch.log10((w**2).sum() / ((w - wq)**2).sum()).item()

def quant_weights(bits, per_channel):
    return [(fq_sym(w, bits, per_channel), b) for w, b in W]

w2 = W[1][0]
s2 = w2.abs().amax(dim=(1, 2, 3)) / 127
ref = torch.fake_quantize_per_channel_affine(w2, s2, torch.zeros(32, dtype=torch.int32), 0, -127, 127)
print("max |fq_sym - torch.fake_quantize_per_channel_affine|:", (fq_sym(w2, 8, True) - ref).abs().max().item())
print(f"FP32 (BN folded) test accuracy: {acc(W):.1f}%")
for i, (w, _) in enumerate(W[:3]):
    r = w.abs().amax(dim=(1, 2, 3))
    print(f"conv{i+1}: largest/smallest channel max|w| = {(r.max()/r.min()).item():.0f}x")

print("\nWeights only (activations FP32), symmetric")
print("bits  granularity   SQNR conv2   accuracy")
for bits in (8, 4):
    for pc in (False, True):
        Wq = quant_weights(bits, pc)
        print(f"W{bits}    {'per-channel' if pc else 'per-tensor ':11s}   "
              f"{sqnr(W[1][0], Wq[1][0]):6.1f} dB    {acc(Wq):5.1f}%")

max |fq_sym - torch.fake_quantize_per_channel_affine|: 0.0
FP32 (BN folded) test accuracy: 95.9%
conv1: largest/smallest channel max|w| = 5x
conv2: largest/smallest channel max|w| = 4x
conv3: largest/smallest channel max|w| = 3x

Weights only (activations FP32), symmetric
bits  granularity   SQNR conv2   accuracy
W8    per-tensor      38.1 dB     96.3%
W8    per-channel     43.4 dB     95.9%
W4    per-tensor      12.8 dB     86.7%
W4    per-channel     18.1 dB     95.0%

The hand-written quantizer agrees with PyTorch’s exactly. After folding, the channel ranges of each layer differ by a factor of 3 to 5, far less than the 100× that Jacob et al. name as a common failure, yet per-channel scales still raise the SQNR of conv2 by 5 dB at either bit width. At 8 bits the weight error is too small to matter: on 1,000 test images, a difference of a few tenths of a point is within sampling noise (one image is 0.1 point, and the binomial standard error near 96% is about 0.6 point). At 4 bits the SQNR falls by 25.3 dB, close to the 25.2 dB that the uniform-error model predicts when the step grows by the factor 127/7, and per-tensor weights lose 9 points where per-channel weights lose under 1, the pattern Krishnamoorthi reports for 4-bit weights.

Next, the activations are quantized too, with weights fixed at INT8 per channel and activation ranges from the three calibration methods, run on 256 training images.

with torch.no_grad():
    taps = forward(xcal, W)[1]
print("\nWeights W8 per-channel; activations affine per-tensor, calibrated on 256 images")
print("act bits   min-max   percentile   mse")
W8 = quant_weights(8, True)
for abits in (8, 4):
    row = [acc(W8, [calibrate(t, abits, m) for t in taps], abits)
           for m in ("min-max", "percentile", "mse")]
    print(f"A{abits}         " + "   ".join(f"{a:6.1f}%" for a in row))

Weights W8 per-channel; activations affine per-tensor, calibrated on 256 images
act bits   min-max   percentile   mse
A8           95.9%     95.9%     95.8%
A4           93.3%     94.7%     95.2%

With 256 levels any of the three ranges is fine. With 16 levels the rounding error dominates, so ranges that clip a few outliers in exchange for a finer step do better than min-max, as in Nagel et al.’s Table 2 at 4 bits.

The integer-only path of Jacob et al. can be checked on the first layer: a uint8 input with a zero point, padding with the zero point (the code of real 0), an int32 accumulator with the zero-point term subtracted, an int32 bias with scale $S_x S_w$, and rescaling by the fixed-point $M_0$ and a rounding shift, with the ReLU as a clamp.

# ---------------------------------------------------------------- integer-only conv1
s_w = W[0][0].abs().amax(dim=(1, 2, 3)) / 127               # per-channel weight scales
qw = torch.clamp(torch.round(W[0][0] / s_w[:, None, None, None]), -127, 127).long()
s_x, z_x = calibrate(taps[0], 8, "min-max")                 # input image quantizer
with torch.no_grad():
    y_cal = F.relu(F.conv2d(fq_aff(taps[0], s_x, z_x, 8), fq_sym(W[0][0], 8, True), W[0][1], padding=1))
s_y, z_y = calibrate(y_cal, 8, "min-max")                   # conv1 output quantizer
x = xte[:100]
qx = torch.clamp(torch.round(x / s_x) + z_x, 0, 255).long()
qx = F.pad(qx, (1, 1, 1, 1), value=z_x)                     # pad with the code of real 0
acc32 = F.conv2d(qx.double(), qw.double()).long()           # sum q_w * q_x (exact)
acc32 = acc32 - z_x * qw.sum(dim=(1, 2, 3))[None, :, None, None]
qb = torch.round(W[0][1] / (s_x * s_w)).long()              # int32 bias, scale S_x*S_w
acc32 = acc32 + qb[None, :, None, None]
M = (s_x * s_w / s_y).double()                              # real multiplier, in (0, 1)
n = torch.floor(-torch.log2(M)).long()                      # M = 2^-n * M0, M0 in [0.5, 1)
M0 = torch.round(M * 2.0 ** n * 2**31).long()               # int32 fixed-point M0
shift = (31 + n)[None, :, None, None]
prod = acc32 * M0[None, :, None, None]
q_int = ((prod + (1 << (shift - 1))) >> shift) + z_y        # rounding right shift
q_int = torch.clamp(q_int, z_y, 255)                        # ReLU and saturate to uint8
with torch.no_grad():
    y_fq = F.relu(F.conv2d(fq_aff(x, s_x, z_x, 8), fq_sym(W[0][0], 8, True),
                           qb * s_x * s_w, padding=1))
q_fq = torch.clamp(torch.round(y_fq / s_y) + z_y, 0, 255).long()
d = (q_int - q_fq).abs()
q_exact = torch.clamp(torch.round(acc32.double() * M[None, :, None, None]) + z_y, z_y, 255)
print(f"\nInteger-only conv1: input zero point {z_x}, M in [{M.min():.4f}, {M.max():.4f}], "
      f"accumulator |max| {acc32.abs().max().item()}")
print(f"outputs differing from fake-quant: {(d > 0).sum().item()} of {d.numel()}, max diff {d.max().item()} LSB")
print(f"outputs differing from exact rescaling of the int32 accumulator: {(q_int != q_exact).sum().item()}")

Integer-only conv1: input zero point 128, M in [0.0012, 0.0058], accumulator |max| 91690
outputs differing from fake-quant: 3 of 1638400, max diff 1 LSB
outputs differing from exact rescaling of the int32 accumulator: 0

Inputs in $[-1, 1]$ give a zero point of 128, and every multiplier $M$ falls in $(0, 1)$, as Jacob et al. observed. The largest accumulator value, 91,690, fits in 18 signed bits, well within int32. The integer layer reproduces the floating-point simulation for all but 3 of 1.6 million outputs, each off by one level. The fixed-point $M_0$ and the shift are not the cause: the integer result equals an exact real-valued rescaling of the int32 accumulator everywhere, so the three differences come from float32 rounding in the simulation near level boundaries. (The shift rounds ties upward, the rule Jacob et al. warn against below; here it made no difference.) This agreement is what lets fake quantization predict the accuracy of the deployed model. Deployment tools rely on it: in TensorRT 11.4.1, fake-quantization operations exported to ONNX become pairs of Quantize and Dequantize nodes that carry the calibrated scales, and the implicit INT8 calibrator classes of earlier versions were removed in TensorRT 11 in favour of these explicit nodes (as of October 2026).

Finally, both weights and activations go to 4 bits (W4A4), the case where post-training quantization struggles, and the network is fine-tuned with fake quantization and the straight-through estimator. Batch normalization stays statically folded, the weight ranges follow the current weights, the activation ranges stay at their MSE calibration, and the learning rate anneals from a tenth of the training rate (our choice).

# ---------------------------------------------------------------- QAT at W4A4
class FQSTE(torch.autograd.Function):
    """Fake quantization with the straight-through estimator: identity
    gradient inside the clipping range, zero outside."""
    @staticmethod
    def forward(ctx, x, s, z, qmin, qmax):
        q = torch.round(x / s) + z
        ctx.save_for_backward((q >= qmin) & (q <= qmax))
        return (torch.clamp(q, qmin, qmax) - z) * s
    @staticmethod
    def backward(ctx, g):
        (inside,) = ctx.saved_tensors
        return g * inside, None, None, None, None

def qat_forward(x, params, act, bits=4):
    qmax = 2 ** (bits - 1) - 1
    def wq(w):                                   # per-channel symmetric, range from w
        s = (w.abs().amax(dim=tuple(range(1, w.dim())), keepdim=True) / qmax).detach()
        return FQSTE.apply(w, s, 0, -qmax, qmax)
    def aq(t, i):
        s, z = act[i]
        return FQSTE.apply(t, s, z, 0, 2 ** bits - 1)
    (w1, b1), (w2, b2), (w3, b3), (w4, b4) = params
    x = aq(x, 0)
    x = F.max_pool2d(F.relu(F.conv2d(x, wq(w1), b1, padding=1)), 2); x = aq(x, 1)
    x = F.max_pool2d(F.relu(F.conv2d(x, wq(w2), b2, padding=1)), 2); x = aq(x, 2)
    x = F.relu(F.conv2d(x, wq(w3), b3, padding=1)).mean((2, 3));     x = aq(x, 3)
    return F.linear(x, wq(w4), b4)

act4 = [calibrate(t, 4, "mse") for t in taps]      # fixed during fine-tuning
print(f"\nW4A4 per-channel, PTQ (MSE calibration): {acc(quant_weights(4, True), act4, 4):.1f}%")
params = [(w.clone().requires_grad_(), b.clone().requires_grad_()) for w, b in W]
opt = torch.optim.Adam([p for pair in params for p in pair], 3e-4)  # 10% of the FP32 rate
sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, 3 * 47)
for epoch in range(3):
    perm = torch.randperm(3000)
    for i in range(0, 3000, 64):
        idx = perm[i:i + 64]
        loss = F.cross_entropy(qat_forward(xtr[idx], params, act4), ytr[idx])
        opt.zero_grad(); loss.backward(); opt.step(); sched.step()
    with torch.no_grad():
        a = (qat_forward(xte, params, act4).argmax(1) == yte).float().mean().item() * 100
    print(f"W4A4 per-channel, QAT epoch {epoch+1}: {a:.1f}%")

W4A4 per-channel, PTQ (MSE calibration): 93.1%
W4A4 per-channel, QAT epoch 1: 94.2%
W4A4 per-channel, QAT epoch 2: 94.7%
W4A4 per-channel, QAT epoch 3: 94.7%

One epoch of quantization-aware fine-tuning recovers most of the gap between W4A4 post-training quantization and FP32, and three epochs leave the network about 1 point below its floating-point accuracy, with every weight on a 15-level grid and every activation on a 16-level one.

The script takes the seed as an argument. Seed 0, shown, gave the lowest FP32 accuracy of five runs (95.9%, against 97.4–98.8% for seeds 1–4). In all five runs (seeds 0–4), INT8 weights, alone at either granularity or with INT8 activations under any of the three calibrations, stayed within 0.4 point of FP32; per-channel scales raised the 4-bit SQNR of conv2 by 3.7 to 6.1 dB; min-max calibration gave the lowest A4 accuracy of the three methods; and QAT finished 0.9 to 2.3 points below FP32, above W4A4 post-training quantization in every run. The 4-bit post-training results varied widely: 65.7% to 94.6% for W4A4, and 49.9% to 93.3% for per-tensor W4 weights. With seed 1, per-tensor W4 weights beat per-channel ones (93.3% against 92.1%) despite their lower SQNR; with seed 3 both failed (49.9% and 68.6%) while QAT reached 96.5%. A lower weight error does not guarantee higher accuracy, and at 4 bits post-training results depend on the particular network.

The formats the hardware offers can be inspected with torch.finfo and torch.iinfo; the second script prints them and the error of storing one row of an intensity ramp in each.

import torch

print("dtype            bits  exp  man     max        smallest normal  eps")
for dt, e, m in [(torch.float32, 8, 23), (torch.float16, 5, 10), (torch.bfloat16, 8, 7),
                 (torch.float8_e4m3fn, 4, 3), (torch.float8_e5m2, 5, 2)]:
    fi = torch.finfo(dt)
    print(f"{str(dt)[6:]:16s} {fi.bits:3d}  {e:3d}  {m:3d}  {fi.max:11.4g}  {fi.smallest_normal:13.4g}  {fi.eps:9.4g}")
for dt in (torch.int8, torch.uint8):
    ii = torch.iinfo(dt)
    print(f"{str(dt)[6:]:16s} {ii.bits:3d}   integers {ii.min} .. {ii.max}")

# The same image row in each format: relative error grows as mantissa bits shrink.
x = torch.linspace(0, 1, 9) * 3.7 + 0.01                      # pixel-like values
for dt in (torch.float16, torch.bfloat16, torch.float8_e4m3fn, torch.float8_e5m2):
    y = x.to(dt).float()
    print(f"{str(dt)[6:]:16s} max rel. error {((y - x).abs() / x).max().item():.4f}")

dtype            bits  exp  man     max        smallest normal  eps
float32           32    8   23    3.403e+38      1.175e-38  1.192e-07
float16           16    5   10     6.55e+04      6.104e-05  0.0009766
bfloat16          16    8    7     3.39e+38      1.175e-38   0.007812
float8_e4m3fn      8    4    3          448        0.01562      0.125
float8_e5m2        8    5    2    5.734e+04      6.104e-05       0.25
int8               8   integers -128 .. 127
uint8              8   integers 0 .. 255
float16          max rel. error 0.0003
bfloat16         max rel. error 0.0024
float8_e4m3fn    max rel. error 0.0312
float8_e5m2      max rel. error 0.0772

BF16 has FP32’s range with FP16’s size but a step eight times coarser than FP16’s; E4M3 trades range for precision against E5M2, and its maximum of 448 and smallest normal of $2^{-6}$ match Micikevicius et al.’s Table 1. Within its normal range, a floating-point format bounds the relative rounding error by half its eps, while an integer grid bounds the absolute error by half its scale (standard properties of rounding to nearest). As of October 2026, the PyTorch 2.14 documentation directs quantization work to the separate torchao library and asks users of the torch.ao.quantization flows to migrate; the example avoids both by writing the quantizers out.

Variants

• Weight-only quantization stores weights in INT8 or INT4 and dequantizes them before floating-point arithmetic. It shrinks the model without calibration data (Krishnamoorthi, §3.1.1); TensorRT 11.4.1 supports INT4 only this way.
• Post-training refinements improve PTQ without retraining. Cross-layer equalization rescales consecutive layers through the positive scaling equivariance of ReLU, so that per-tensor quantization of MobileNetV2 rises from 0.12% to 69.91% ImageNet accuracy (Nagel et al., Table 3). Bias correction removes the mean shift that quantization error causes. AdaRound learns whether to round each weight up or down, using a small calibration set (Nagel et al., §3.2–3.4).
• Mixed precision keeps sensitive layers at higher precision. Wu et al.’s partial quantization leaves the layers whose quantization costs the most accuracy, found one at a time, in floating point. Searches over bit widths per layer use reinforcement learning with feedback from a hardware simulator, the trace of the Hessian as a sensitivity measure (HAWQ), or integer linear programming (HAWQv3), which Gholami et al. report gave up to 50% speedup with INT4/INT8 over INT8 on T4 GPUs.
• Dyadic quantization generalizes power-of-two scales to scales of the form integer over a power of two, so that inference needs no division (Gholami et al., §IV-A).
• Non-uniform quantization places levels unevenly, for example by k-means on the weights, which compresses further but needs a lookup before arithmetic, so it saves storage and bandwidth rather than computation (Krishnamoorthi; Wu et al., §2).
• Binary and ternary networks quantize weights, and sometimes activations, to one or two bits. Jacob et al. note that 1-bit quantization often costs substantial accuracy, and that bit-shift multiplications help little on hardware with pipelined multiply-add instructions.

Best Practices

The sources converge on a few recommendations for convolutional networks:

• Symmetric per-channel weights and per-tensor activations. Krishnamoorthi recommends symmetric per-channel weight quantization as the starting point for PTQ and asks accelerators to support per-channel weights. Wu et al. recommend symmetric per-channel weights in $[-127, 127]$ with max calibration and per-tensor scale-only activations. Nagel et al. recommend symmetric weights with affine activations, and per-channel weights if the target supports them.
• Try several calibrations. Wu et al. found no single activation calibration best for every network and recommend running max, entropy, 99.99% and 99.999% and keeping the best. Nagel et al. recommend MSE range setting, with cross-entropy for the logits.
• Match training to inference. Fake quantization must sit exactly where the inference engine quantizes and must model the operator fusion it performs: no quantizer between an addition and the ReLU it is fused with (Krishnamoorthi, §3.2.1).
• Start QAT from a trained floating-point model. Krishnamoorthi found fine-tuning a floating-point checkpoint more accurate than training quantized from scratch. Wu et al. fine-tune for about 10% of the original schedule, with an annealing learning rate starting at 1% of the initial one. Nagel et al. initialize QAT from PTQ ranges, which helps early in training, though the advantage mostly vanishes later (their Table 8).
• Debug layer by layer. Nagel et al. start by checking that the quantized model matches FP32 with quantization disabled, then quantize weights and activations separately, then one layer at a time, as their flow chart describes.

Pitfalls

• Per-tensor weights after folding. Krishnamoorthi’s per-layer INT8 PTQ of MobileNet-v1 reached 0.1% ImageNet top-1, and Wu et al. found that folding before per-tensor INT8 weight quantization dropped EfficientNet b0 from 76.68% to 12.93%; per-channel scales avoided both (their Table 3).
• Outliers under min-max calibration. One large activation stretches the range of a tensor. Max calibration gave 0.12% top-1 for Inception v4 and 22.3% for EfficientNet b0 in Wu et al.’s Table 5, against 79.60% and 72.06% with entropy calibration.
• Clipping too much. Percentile calibration at 99.9% clipped too aggressively and lowered accuracy on most networks in the same table, for example to 26.18 F1 for BERT Large against 89.59 at 99.99%.
• Unbounded activations. Krishnamoorthi attributes the small loss from 8-bit activations in MobileNet-v1 partly to ReLU6, which fixes activations to $[0, 6]$. Jacob et al. found the same, with less degradation for ReLU6 than for ReLU in quantized Inception v3. In QAT, however, Krishnamoorthi got slightly better accuracy with ReLU, letting training set the ranges.
• Averaged weights in QAT. Krishnamoorthi warns that exponential moving averages of the weights can hurt QAT, because small differences between averaged and current weights flip their quantized values. Folding batch normalization with per-batch statistics causes a similar jitter (see Quantization-aware training).
• Biased rounding. In Appendix B of the arXiv version, Jacob et al. warn that a rounding shift that breaks ties upward, as the ARM NEON instruction RSHL does, biases results upward enough to cost end-to-end accuracy.
• Mismatch between simulation and device. A simulation that treats average pooling, additions or concatenations differently from the device will not predict its accuracy, and an activation such as Swish that the hardware cannot fuse needs extra quantizers around it (Nagel et al., §2.3.2–2.3.3).
• Assuming a speedup. A smaller model is not faster unless the runtime has kernels for the format on the target. Wu et al. warn that the input-dependent zero-point term of affine weights can reduce or eliminate the throughput gain of integer arithmetic.

Evidence and Impact

Reported costs depend on the network, the bit widths and the method:

• Jacob et al. (2018), ImageNet and COCO. Integer-only ResNets of depth 50, 100 and 150 reached 74.9%, 76.6% and 76.7% ImageNet accuracy against 76.4%, 78.0% and 78.8% in floating point (Table 4.1). The text says within 2%, but the depth-150 gap in the table is 2.1 points. Quantized MobileNet SSD on COCO kept 21.7 mAP against 22.1, with single-core Snapdragon 835 latency of 272 ms against 370 ms on a big core (Table 4.4, depth multiplier 100%). For face attributes, 8-bit weights lost 0.3–3.5% relative average precision across activation bit widths of 8 down to 4, and 4-bit weights 10.8–14.0% (Table 4.6). At the 33 ms needed for 30 fps on Snapdragon 835 LITTLE cores, integer MobileNets were about 10% more accurate than floating-point MobileNets of the same latency.
• Krishnamoorthi (2018), ImageNet top-1. The abstract says that 8-bit per-channel weights with per-layer activations after training come within 2% of floating point and that QAT narrows the gap to 1%. The tables show slightly larger gaps: up to 2.2 points after training (MobileNet-v2, 0.697 against 0.719; Table 3) and up to 1.8 points after QAT (ResNet-v2-152; Table 4). QAT also made per-layer schemes work: MobileNet-v1 0.70 against 0.709 in floating point (Table 4). With 4-bit per-channel weights, QAT reached 0.65 for MobileNet-v1 and 0.732 for ResNet-v1-50, against PTQ 0.001 and 0.54 (Table 5). Measured on one big core of a Pixel 2, quantized models ran 2 to 3 times faster than floating point, and nearly 10 times faster on a Qualcomm DSP (Table 7).
• Nagel et al. (2021), ImageNet, Pascal VOC and COCO. The text says their PTQ pipeline kept W8A8 within 0.7% of FP32 for all models; in Table 6 the loss is at most 0.73 points for the ImageNet classifiers, DeepLabV3 and BERT-base, but 1.4 to 1.8 mAP for EfficientDet-D1. At W4A8, EfficientDet-D1 fell from 40.08 to 0.31 mAP with per-tensor weights and to 35.08 with per-channel. QAT at W4A8 brought ResNet18 to 70.01% against 69.68% FP32 with per-channel weights, while W4A4 still cost MobileNetV2 nearly 5 points (66.89% against 71.72%; Table 10).
• Wu et al. (2020), 19 models. With max-calibrated per-channel weights and the best of their activation calibrations, INT8 PTQ stayed within 1% relative of FP32 except for the MobileNets, EfficientNets, Transformer and BERT. INT8 QAT brought every network within 1% relative, for example EfficientNet b0 from 72.06% after PTQ to 76.95% against 76.85% (Table 7).
• FP8 (Micikevicius et al., 2022). Training with FP8 inputs to every convolution and matrix product matched 16-bit training within run-to-run variation for the image classifiers they tested on ImageNet, except MobileNet v2 (71.04% against 71.65%; Table 2).