LeNet

LeNet is the family of convolutional neural networks that Yann LeCun and colleagues at AT&T Bell Laboratories developed between 1989 and 1998 to read handwritten digits and characters. The networks take a size-normalized image of one character and output its class. Their design, convolutional layers with shared weights alternating with subsampling layers and followed by fully connected layers, all trained end to end by backpropagation from pixels, is the pattern later CNNs kept. The names LeNet-1, LeNet-4 and LeNet-5 are those used by LeCun, Bottou, Bengio and Haffner (1998), whose LeNet-5 is the version usually meant by “LeNet” today. LeNet-5 has seven layers with trainable parameters, about 60,000 parameters in all, and reached 0.95% error on the MNIST test set, 0.8% when trained with distorted images.

Problem Addressed

The task was isolated character recognition for postal and banking documents. The usual method of the time was a hand-designed feature extractor followed by a trainable classifier; LeCun et al. (1998, §II) argue that a fully connected network fed with raw pixels instead has too many weights for the available data and no built-in invariance to the shifts and distortions that imperfect size normalization leaves in characters, and that it ignores the 2D topology of the image entirely. The zip-code network of LeCun et al. (1989, Neural Computation) had already taken the approach the 1998 paper develops: build the prior knowledge into the architecture as constraints on the weights (local receptive fields, weights shared across positions, reduced resolution in later layers), and learn everything else from the pixels. The authors call this “constrained backpropagation” and contrast it with their group’s previous system, whose first convolutional stages used hand-designed kernels.

Architecture

LeNet-5 maps a $32 \times 32$ grayscale image to ten class scores through three convolutional layers (C1, C3, C5), two subsampling layers (S2, S4), a fully connected layer (F6) and an output layer of radial basis function (RBF) units. The layer names are the paper’s: C for convolution, S for subsampling, F for fully connected, and the number is the layer index.

Layer	Operation	Output	Trainable parameters	Connections
Input	size-normalized character	$1 \times 32 \times 32$
C1	6 kernels, $5 \times 5$	$6 \times 28 \times 28$	156	122,304
S2	$2 \times 2$ sum, coefficient and bias per map	$6 \times 14 \times 14$	12	5,880
C3	16 kernels, $5 \times 5$, partial connection table	$16 \times 10 \times 10$	1,516	151,600
S4	as S2	$16 \times 5 \times 5$	32	2,000
C5	120 kernels, $5 \times 5$ over all 16 maps	$120 \times 1 \times 1$	48,120	48,120
F6	fully connected	84	10,164	10,164
Output	10 Euclidean RBF units with fixed centres	10

The counts are those printed in LeCun et al. (1998, §II.B); the paper gives the network’s totals as 340,908 connections and 60,000 trainable parameters. The first count includes the output layer’s 840 connections and the second does not, as the code below confirms by recomputing both.

Input

The input is larger than the characters. MNIST digits fit in a $20 \times 20$ box centred in a $28 \times 28$ field, and LeNet-5 sees them padded to $32 \times 32$ with background pixels. The paper’s reason is that distinctive features such as stroke end-points and corners should be able to fall at the centre of the receptive fields of the highest feature detectors; the centres of the C3 units’ receptive fields cover a $20 \times 20$ area in the middle of the input. Pixels are scaled so that the white background is $-0.1$ and black ink $1.175$, which makes the input mean roughly 0 and its variance roughly 1.

Convolutional layers C1, C3 and C5

Each unit of a feature map sees a $5 \times 5$ neighbourhood and shares its 25 weights and bias with every other unit of the map, so C1, with no padding, turns the $32 \times 32$ input into six $28 \times 28$ maps.

C3 does not connect every S2 map to every C3 map. Its Table I assigns each of the 16 maps a subset of the six S2 maps: the first six take every contiguous triple (counting cyclically, as the table shows), the next six every contiguous four, the next three some non-contiguous fours, and the last one all six. That gives 60 map-to-map connections instead of 96, hence $60 \cdot 25 + 16 = 1{,}516$ parameters. The paper gives two reasons: the scheme keeps the number of connections within bounds, and, more importantly, it breaks the symmetry of the network, so that maps receiving different inputs are pushed to extract different, hopefully complementary features.

C5 has 120 maps, each unit seeing a $5 \times 5$ window in all 16 S4 maps. On a $32 \times 32$ input its maps are $1 \times 1$, so it amounts to a full connection; the authors still call it convolutional because a larger input would make its maps larger than $1 \times 1$, and point to §VII for networks enlarged this way. In our reading, this is the property that fully convolutional networks later exploited.

Trainable subsampling S2 and S4

Each S2 unit adds the four values of a non-overlapping $2 \times 2$ window of one C1 map, multiplies the sum by a trainable coefficient, adds a trainable bias and passes the result through the squashing function, so each map has two parameters:

$$x = f\Big(w \sum_{(u,v) \in 2 \times 2} c_{uv} + b\Big).$$

Section II.A describes the same unit as taking the average of its four inputs; the two descriptions differ only by a factor of 4 that the coefficient absorbs. The paper explains the coefficient’s role: when it is small the unit is nearly linear and the layer merely blurs, and when it is large the unit acts as a “noisy OR” or “noisy AND” depending on the bias. The pooling article covers why subsampling lowers sensitivity to small shifts and what it costs.

Squashing function

Every layer up to F6 computes a weighted sum $a$ plus a bias and applies a scaled hyperbolic tangent (LeCun et al., 1998, Eq. 6 and Appendix A):

$$f(a) = A \tanh(S a), \qquad A = 1.7159, \quad S = \tfrac{2}{3}.$$

With these constants $f(1) = 1$ and $f(-1) = -1$, the overall gain is about 1 in normal operation, and $|f''|$ is largest at $\pm 1$, which the authors say helps convergence towards the end of training. They describe the choice as a convenience that does not affect the result. Weights start uniform in $\pm 2.4/F_i$, where $F_i$ is the fan-in of the unit, to keep the initial weighted sums in the operating range of the squashing function (the weight initialization article covers the later, variance-based rules).

F6 and the RBF output layer

F6 has 84 units, a number set by the output layer. Each of the ten output units computes the squared Euclidean distance between the F6 vector $\mathbf{x}$ and its own centre $\mathbf{w}_i$:

$$y_i = \sum_{j=1}^{84} (x_j - w_{ij})^2 .$$

The output is a penalty, low for the correct class. The centres were chosen by hand and kept fixed, “at least initially”: each is a stylized image of its character drawn on a $7 \times 12$ bitmap, with components $+1$ or $-1$. The paper gives three reasons for such distributed codes over one unit per class: confusable characters (O, o and 0; l, 1 and I) get similar codes, which helps a linguistic post-processor choose among them in the full ASCII version; one-of-N codes behave badly when there are more than a few dozen classes, because sigmoid units would have to be off most of the time; and RBF units respond only in a bounded region of their input space, which makes rejecting non-characters easier. Targets of $\pm 1$ sit where the squashing function’s curvature is largest, so F6 works in its most nonlinear range without saturating.

Training

LeNet-5 was trained on the 60,000 MNIST training images, which the authors built from NIST Special Databases 1 and 3, putting the first 250 of SD-1’s writers in the training set and the other 250 in the test set (§III.A). The simplest loss pushes down the penalty of the correct class $D_p$ for each pattern $Z^p$, which the paper notes is equivalent to a mean squared error; it lacks competition between classes, and if the centres were allowed to adapt, all of them could collapse onto one point. The authors instead use a MAP criterion with a “rubbish” class whose penalty is a positive constant $j$ (Eq. 9):

$$E(W) = \frac{1}{P} \sum_{p=1}^{P} \Big( y_{D_p}(Z^p, W) + \log\big(e^{-j} + \textstyle\sum_i e^{-y_i(Z^p, W)}\big) \Big),$$

which also pulls up the penalties of the incorrect classes. With weight sharing, the gradient for a shared weight is the sum of the gradients of all the connections that use it (Eq. 19).

The optimizer was not plain stochastic gradient descent but a stochastic diagonal Levenberg–Marquardt method (Appendix C): after every single pattern, each parameter $w_k$ takes a step with its own learning rate

$$\epsilon_k = \frac{\eta}{\mu + h_{kk}},$$

where $h_{kk}$ is a Gauss–Newton estimate of the second derivative of the loss with respect to $w_k$, re-estimated on 500 patterns before each pass, and $\mu = 0.02$ guards against tiny curvature estimates. The authors argue that weight sharing makes the loss ill-conditioned, since one early-layer weight can have a large effect on the output, and that per-parameter rates compensate. Over 20 passes, $\eta$ fell in steps from 0.0005 to 0.00001. The test error stabilized at 0.95% after about 10 passes, the training error reached 0.35% after 19, and no overtraining was observed. Training took 2 to 3 days on one 200 MHz R10000 processor of a Silicon Graphics Origin 2000 server (§III.D).

To test whether more data would help, the authors added 540,000 randomly distorted copies of the training images (planar affine combinations of translation, scaling, squeezing and horizontal shearing); with the same 20 passes, the test error fell to 0.8% (§III.B).

Inference

A forward pass gives ten penalties, and the class is the one with the lowest. In applications the network must also reject: in the 1998 comparison, a pattern is rejected when the difference between the scores of the two best classes is below a threshold. LeNet-5 needs about 401,000 multiply-accumulate operations to classify one size-normalized character and stores about 60,000 variables, against about 14 million for a degree-4 polynomial support vector machine with 1.1% error (Figures 11 and 12, whose values are in thousands, as the text’s 650,000 multiply-adds for the reduced-set SVM confirms).

The authors estimate that LeNet-5 recognizes characters correctly under scale changes up to about a factor of 2, vertical shifts of about half the character height and rotations up to about $\pm 30^\circ$, and show it reading characters in heavy salt-and-pepper noise after training on MNIST with each pixel inverted with probability 0.1 (§III.E).

Performance

All results below are test error rates on the 10,000 MNIST test images, as LeCun et al. (1998, Figure 9) report them; the caption puts the uncertainty at about 0.1%. [dist] marks training with the distorted images, [16×16] the low-resolution version of the database.

Classifier	Test error	Multiply-accumulates per character	Stored variables
Linear classifier	12.0%	4,000	4,000
K-nearest neighbours, Euclidean	5.0%
Fully connected, 300 hidden units	4.7%
Fully connected, 500–150 hidden units	2.95%	469,000	469,000
SVM, polynomial of degree 4	1.1%	14 million	14 million
[dist] Virtual SVM, degree 9	0.8%	28 million	28 million
[16×16] LeNet-1	1.7%	100,000	3,000
LeNet-4	1.1%	260,000	17,000
LeNet-5	0.95%	401,000	60,000
[dist] LeNet-5	0.8%	401,000	60,000
[dist] Boosted LeNet-4	0.7%	460,000	51,000

Operation and memory counts are read from Figures 11 and 12, in thousands; blank cells are not given for that exact variant. Figures 9, 11 and 12 label the 2.95% network 28×28-500-150-10; the text (§III.C.6) calls it 28×28-1000-150-10. Figures 9 and 11 label the virtual SVM “poly 9”, Figure 12 “poly 5”. The authors also stress training-set size: LeNet-5’s test error falls as the training set grows from 15,000 to 60,000 images, which they read as evidence that more data would still help (Figure 6).

Strengths

• Few parameters for its size. Weight sharing gives about 60,000 parameters for 340,908 connections, and the authors argue that the smaller capacity narrows the gap between training and test error (§II.A).
• Learned features. The network synthesizes its own feature extractor from pixels, and in the comparison it beat the memory-based classifiers (nearest neighbours, tangent distance) with a small fraction of their memory and computation (§III.D).
• Robustness to distortion and noise, as estimated above, from the convolution-subsampling structure.
• Suited to hardware. The regular structure and small weight memory made LeNets easy to implement; single-chip analog-digital implementations of LeNet-5’s predecessors ran at over 1,000 characters per second (§III.D).

Limitations

• Fixed-size, pre-segmented input. LeNet-5 classifies one size-normalized character at a time, so a full system needs field location and segmentation around it, which the 1998 paper builds with graph transformer networks; its space displacement neural network, which replicates the recognizer across the input, is presented in the conclusions as a way to avoid explicit segmentation.
• Hand-designed parts. The C3 connection table and the RBF output codes are engineered choices; both are dropped by modern re-implementations.
• Small scale and slow training by modern standards. Days of CPU time for a 60,000-parameter network on 60,000 images; the authors write that in 1989 a LeNet-5-sized recognizer would have needed several weeks of training and more data than was available.
• Data hunger. Even on MNIST, the network’s error was still falling with more training data (Figure 6). Classifying natural photographs at scale needed much larger networks, data and compute, which arrived with AlexNet in 2012.

Historical Importance

The zip-code network (1989). LeCun et al. (1989, Neural Computation) trained a network on 9,298 handwritten zip-code digits from U.S. mail passing through the Buffalo, NY post office, normalized to $16 \times 16$ pixels with gray levels in $[-1, 1]$ (7,291 for training, 2,007 for testing). It has no separate subsampling layers: layer H1 has 12 feature maps of $8 \times 8$ units, each seeing a $5 \times 5$ window of the input, with neighbouring units two pixels apart, and H2 has 12 maps of $4 \times 4$, each unit seeing $5 \times 5$ windows in 8 of the 12 H1 maps. Kernels are shared within a map, but biases are not, so H1 has 768 biases plus 300 kernel weights. Two fully connected layers of 30 and 10 units follow, for 1,256 units, 64,660 connections and 9,760 parameters. Trained with stochastic (per-pattern) updates using a diagonal approximation of the Hessian, a mean squared error loss and scaled tanh units, it reached 0.14% training and 5.0% test error after 23 passes, and needed 12.1% of test patterns rejected to reach 1% error on the rest; a fully connected network with 40 hidden units got 8.1% test error and 19.4% rejections. The authors cite Rumelhart, Hinton and Williams (1986) as describing weight sharing, for the T-C problem, and present the work as carrying LeCun’s earlier architecture-design experiments on simple digit images over to a real-world task.

Separate averaging layers (1989–1990). In the NIPS 2 paper (LeCun et al., 1989, NIPS), each feature extraction layer is followed by a layer that performs local averaging and subsampling, which the authors liken to the neocognitron. The network takes a $16 \times 16$ image centred in a $28 \times 28$ input plane and has four $24 \times 24$ maps, four $12 \times 12$ averaging maps, 12 maps of $8 \times 8$ connected to one or two averaging maps each, 12 maps of $4 \times 4$ and 10 outputs: 4,635 units, 98,442 connections and 2,578 parameters, obtained by pruning a network with four times as many parameters with Optimal Brain Damage. Trained on the 7,291 zip-code digits plus 2,549 printed ones, it had 3.4% error on the 2,007 handwritten and 700 printed test digits, and needed 9% rejections for 1% error on the handwritten ones. The 1998 paper’s LeNet-1 has the same input arrangement and “about 2600” parameters and cites this paper for its development, so we take them to be the same network.

The neocognitron. Fukushima’s neocognitron (1980) is the precursor both 1989 papers and the 1998 paper name. LeCun et al. (1998), citing Fukushima and Miyake’s 1982 version, describe it as implementing the same combination of convolution and subsampling, inspired by Hubel and Wiesel’s simple and complex cells, at a time when no globally supervised learning procedure such as backpropagation was available; the NIPS paper names backpropagation instead of unsupervised learning as the notable difference. The receptive field article describes the neocognitron’s cells.

Deployment (1996). In the 1998 paper’s check reading system, LeNet-5 is the character recognizer inside a graph transformer network, classifying segment images into the 95 printable ASCII characters plus a rubbish class; it was trained first on 500,000 character images and then on characters segmented from checks. On 646 machine-printed business checks the system read 82% correctly with 1% errors and 17% rejects, against 68%, 1% and 31% for the previous system. The authors report that it was integrated into NCR’s check reading systems, fielded in several U.S. banks from June 1996, and reading millions of checks per day (§X.D).

LeNet-1 was, in the authors’ words, considered the state of the art for a long time, and several of the alternatives in the 1998 comparison (tangent distance, optimal margin classifiers, local learning) were developed to improve on it.

Variants

• LeNet-1 (1989–1990): about 2,600 parameters and about 100,000 multiply-adds in the 1998 paper (four first-layer maps and 2,578 parameters if it is the NIPS network, as argued above); 1.7% MNIST test error on $16 \times 16$ images.
• LeNet-4: four first-level maps, eight subsampling maps connected in pairs to them, 16 feature maps, 16 subsampling maps, a fully connected layer of 120 units and 10 outputs; about 260,000 connections and 17,000 parameters, 1.1% test error. Replacing its last layer by a nearest-neighbour classifier or by Bottou and Vapnik’s local learning did not lower the raw error but improved rejection (§III.C.8).
• Boosted LeNet-4: three LeNet-4s combined by Drucker et al.’s boosting. The second is trained on a 50/50 mix of patterns the first got right and wrong, the third on patterns where the first two disagree, and the three outputs are added. The distorted images were needed to find enough training patterns, and since the other two nets are skipped when the first is confident, the average cost is about 1.75 times one net. Its 0.7% was the best result in the paper.
• LeNet-5 (1998): the network described above.
• Modern “LeNet-5”. Re-implementations in today’s frameworks keep LeNet-5’s layer sizes but change most of its components; PyTorch’s introductory tutorial (as of October 2026) is an example. The differences:
  • $2 \times 2$ max pooling with no parameters instead of trainable sum-coefficient-bias subsampling;
  • ReLU instead of the scaled tanh;
  • C3 connected to all six maps, giving 2,416 parameters instead of 1,516;
  • C5 written as a linear layer from the flattened $16 \times 5 \times 5$ maps, equivalent for a $32 \times 32$ input but no longer convolutional;
  • a linear output layer of ten logits instead of fixed RBF codes. The usual loss on these is softmax cross-entropy, a modern convention rather than part of the tutorial or the paper;
  • optimizers from torch.optim, such as SGD, instead of per-pattern diagonal Levenberg–Marquardt updates. The result has 61,706 parameters.

Implementations

The PyTorch code below (run with PyTorch 2.14.1) builds LeNet-5 as printed, with the scaled tanh, trainable subsampling, Table I’s connection mask, $\pm 2.4/F_i$ initialization of the convolutional and fully connected layers, fixed $7 \times 12$ RBF codes and the MAP loss, alongside the modern version. It checks every layer’s output shape, parameters and connections against the paper, then trains both for four epochs on synthetic digits: seven-segment-style strokes with random size, slant, position and stroke width, drawn in the central $20 \times 20$ field of a $32 \times 32$ image with the paper’s background and ink levels. The paper does not give a value for $j$, so $j = 5$ is our choice, and plain SGD with momentum replaces the diagonal Levenberg–Marquardt optimizer.

import torch
from torch import nn
import torch.nn.functional as F

torch.manual_seed(0)
torch.set_num_threads(4)

# ---- Synthetic digits: seven-segment strokes with jitter, slant and thickness -------------
#   vertices: 0 top-left, 1 top-right, 2 mid-left, 3 mid-right, 4 bottom-left, 5 bottom-right
SEG = {"a": (0, 1), "b": (1, 3), "c": (3, 5), "d": (4, 5), "e": (2, 4), "f": (0, 2), "g": (2, 3)}
DIGITS = ["abcdef", "bc", "abged", "abgcd", "fgbc", "afgcd", "afgedc", "abc", "abcdefg", "abcdfg"]
BOX = torch.tensor([[0., 0.], [1., 0.], [0., .5], [1., .5], [0., 1.], [1., 1.]])  # (x, y), y down

def render(digit, H, W, x0, y0, w, h, slant, thick, jitter):
    """Anti-aliased strokes of one digit on an H x W canvas; values in [0, 1]."""
    v = BOX + jitter                                         # (6, 2) jittered vertices
    px = x0 + w * (v[:, 0] + slant * (0.5 - v[:, 1]))        # shear: slant the digit
    py = y0 + h * v[:, 1]
    yy, xx = torch.meshgrid(torch.arange(H) + .5, torch.arange(W) + .5, indexing="ij")
    dist = torch.full((H, W), 1e9)
    for s in DIGITS[digit]:
        i, j = SEG[s]
        ax, ay, bx, by = px[i], py[i], px[j], py[j]
        t = (((xx - ax) * (bx - ax) + (yy - ay) * (by - ay)) / ((bx - ax)**2 + (by - ay)**2)).clamp(0, 1)
        dist = torch.minimum(dist, torch.hypot(xx - ax - t * (bx - ax), yy - ay - t * (by - ay)))
    return (thick / 2 + .5 - dist).clamp(0, 1)

def make_digits(n, gen):
    """n random 32x32 digits: a 20x20 field (as in MNIST) centred in 32x32, ink 1.175, paper -0.1."""
    X, y = torch.empty(n, 1, 32, 32), torch.randint(0, 10, (n,), generator=gen)
    for k in range(n):
        r = torch.rand(6, generator=gen)
        h = 14 + 6 * r[0]; w = h * (0.45 + 0.2 * r[1])
        x0 = 6 + (20 - w) * r[2]; y0 = 6 + (20 - h) * r[3]
        img = render(int(y[k]), 32, 32, x0, y0, w, h, slant=0.5 * (r[4] - .5), thick=1.5 + 1.5 * r[5],
                     jitter=0.06 * torch.randn(6, 2, generator=gen))
        X[k, 0] = -0.1 + 1.275 * img
    return X, y

# ---- LeNet-5 as printed in LeCun et al. (1998) -----------------------------------------------
A, S = 1.7159, 2 / 3
def squash(a):                                   # scaled tanh, f(a) = A tanh(S a); f(1) = 1
    return A * torch.tanh(S * a)

TABLE = ["0,1,2", "1,2,3", "2,3,4", "3,4,5", "0,4,5", "0,1,5",                  # C3 columns 0-5
         "0,1,2,3", "1,2,3,4", "2,3,4,5", "0,3,4,5", "0,1,4,5", "0,1,2,5",      # columns 6-11
         "0,1,3,4", "1,2,4,5", "0,2,3,5", "0,1,2,3,4,5"]                        # columns 12-15

class Subsample(nn.Module):
    """Trainable subsampling: sum each 2x2 block, times a coefficient, plus a bias (one each per map)."""
    def __init__(self, maps):
        super().__init__()
        self.coef, self.bias = nn.Parameter(torch.ones(maps) / 4), nn.Parameter(torch.zeros(maps))
    def forward(self, x):
        s = 4 * F.avg_pool2d(x, 2)
        return squash(s * self.coef[:, None, None] + self.bias[:, None, None])

class LeNet5Paper(nn.Module):
    def __init__(self, codes):
        super().__init__()
        self.c1, self.s2 = nn.Conv2d(1, 6, 5), Subsample(6)
        self.c3, self.s4 = nn.Conv2d(6, 16, 5), Subsample(16)
        self.c5, self.f6 = nn.Conv2d(16, 120, 5), nn.Linear(120, 84)
        mask = torch.zeros(16, 6, 1, 1)
        for col, rows in enumerate(TABLE):
            mask[col, [int(r) for r in rows.split(",")]] = 1
        self.register_buffer("mask", mask)       # C3 partial connection table (Table I)
        self.register_buffer("codes", codes)     # fixed +-1 RBF centres, 10 x 84
        with torch.no_grad():                    # weights ~ U(-2.4/F, 2.4/F), F = fan-in of the unit
            for layer, fan in [(self.c1, 25), (self.c5, 400), (self.f6, 120)]:
                layer.weight.uniform_(-2.4 / fan, 2.4 / fan); layer.bias.uniform_(-2.4 / fan, 2.4 / fan)
            fan3 = 25 * mask.sum(dim=(1, 2, 3))  # 75, 100 or 150 inputs per C3 unit
            self.c3.weight.copy_((torch.rand_like(self.c3.weight) * 2 - 1) * (2.4 / fan3)[:, None, None, None])
            self.c3.bias.copy_((torch.rand(16) * 2 - 1) * 2.4 / fan3)
    def features(self, x):
        x = self.s2(squash(self.c1(x)))
        x = self.s4(squash(F.conv2d(x, self.c3.weight * self.mask, self.c3.bias)))
        return squash(self.f6(squash(self.c5(x)).flatten(1)))
    def forward(self, x):                        # RBF outputs y_i = sum_j (x_j - w_ij)^2: lower is better
        return ((self.features(x)[:, None, :] - self.codes) ** 2).sum(-1)

def map_loss(y, target, j=5.0):
    """Eq. (9): penalty of the correct class + log(e^-j + sum_i e^-y_i); j > 0 (value is our choice)."""
    rubbish = torch.full_like(y[:, :1], -j)
    return (y.gather(1, target[:, None])[:, 0] + torch.logsumexp(torch.cat([rubbish, -y], 1), 1)).mean()

# ---- The usual modern re-implementation --------------------------------------------------------
def modern_lenet5():
    return nn.Sequential(nn.Conv2d(1, 6, 5), nn.ReLU(), nn.MaxPool2d(2),
                         nn.Conv2d(6, 16, 5), nn.ReLU(), nn.MaxPool2d(2),     # C3 fully connected
                         nn.Flatten(), nn.Linear(400, 120), nn.ReLU(),        # "C5" as a linear layer
                         nn.Linear(120, 84), nn.ReLU(), nn.Linear(84, 10))    # logits for softmax

# Output codes: each class drawn as a stylized 7-wide x 12-high bitmap, then +1 (ink) / -1.
codes = torch.stack([render(d, 12, 7, 1, 1, 5, 10, 0, 1.6, torch.zeros(6, 2)).flatten() for d in range(10)])
codes = torch.where(codes > 0.5, 1.0, -1.0)
paper = LeNet5Paper(codes)                       # used for the shape check below

# ---- Shapes, parameters and connections versus the paper ---------------------------------------
x = torch.zeros(1, 1, 32, 32)
printed = {"C1": (156, 122_304), "S2": (12, 5_880), "C3": (1_516, 151_600), "S4": (32, 2_000),
           "C5": (48_120, 48_120), "F6": (10_164, 10_164)}
c1 = squash(paper.c1(x)); s2 = paper.s2(c1)
c3 = squash(F.conv2d(s2, paper.c3.weight * paper.mask, paper.c3.bias)); s4 = paper.s4(c3)
c5 = squash(paper.c5(s4)); f6 = paper.features(x)
params = {"C1": 6 * 26, "S2": 12, "C3": int(25 * paper.mask.sum()) + 16, "S4": 32,
          "C5": sum(p.numel() for p in paper.c5.parameters()), "F6": sum(p.numel() for p in paper.f6.parameters())}
# connections = (inputs + bias) per unit x number of units; for C3 the inputs differ per map
conns = {"C1": 26 * c1[0].numel(), "S2": 5 * s2[0].numel(),
         "C3": int(sum((25 * paper.mask[m].sum() + 1) * 100 for m in range(16))),
         "S4": 5 * s4[0].numel(), "C5": 401 * 120, "F6": 121 * 84}
print("layer  output         params (paper)      connections (paper)")
for name, t in zip(printed, [c1, s2, c3, s4, c5, f6]):
    print(f"{name:5}  {str(tuple(t.shape[1:])):13}  {params[name]:6,} ({printed[name][0]:6,})"
          f"   {conns[name]:8,} ({printed[name][1]:8,})")
print(f"RBF    (10,)          {codes.numel():6,} fixed       {codes.numel():8,}")
print(f"total trainable {sum(params.values()):,}; connections incl. RBF {sum(conns.values()) + codes.numel():,}"
      f"  (paper: 60,000 and 340,908)")
print(f"modern re-implementation: {sum(p.numel() for p in modern_lenet5().parameters()):,} parameters")

# ---- A short training run on synthetic digits ---------------------------------------------------
gen = torch.Generator().manual_seed(1)
X_tr, y_tr = make_digits(6000, gen)
X_te, y_te = make_digits(2000, gen)
print(f"\ntraining image 0 (class {y_tr[0]}), central 20x20 field  |  its fixed 7x12 output code")
for r in range(20):
    ink = "".join("#" if v > 0.6 else "+" if v > 0.2 else "." for v in X_tr[0, 0, 6 + r, 6:26])
    code = "".join("#" if v > 0 else "." for v in codes[y_tr[0]].view(12, 7)[r]) if r < 12 else ""
    print(f"  {ink}    {code}")

def run(net, loss_fn, predict, lr, seed, epochs=4):
    opt = torch.optim.SGD(net.parameters(), lr=lr, momentum=0.9)
    order = torch.Generator().manual_seed(seed)
    for _ in range(epochs):
        for b in torch.randperm(len(y_tr), generator=order).split(32):
            loss = loss_fn(net(X_tr[b]), y_tr[b])
            opt.zero_grad(); loss.backward(); opt.step()
    with torch.no_grad():
        return (predict(net(X_te)) != y_te).float().mean().item()

for seed in range(3):
    torch.manual_seed(seed)
    e_paper = run(LeNet5Paper(codes), map_loss, lambda y: y.argmin(1), lr=0.003, seed=seed)
    torch.manual_seed(seed)
    e_modern = run(modern_lenet5(), F.cross_entropy, lambda z: z.argmax(1), lr=0.03, seed=seed)
    print(f"seed {seed}: test error, paper-style (nearest RBF code) {e_paper:.2%},"
          f" modern (largest logit) {e_modern:.2%}")

Output:

layer  output         params (paper)      connections (paper)
C1     (6, 28, 28)       156 (   156)    122,304 ( 122,304)
S2     (6, 14, 14)        12 (    12)      5,880 (   5,880)
C3     (16, 10, 10)    1,516 ( 1,516)    151,600 ( 151,600)
S4     (16, 5, 5)         32 (    32)      2,000 (   2,000)
C5     (120, 1, 1)    48,120 (48,120)     48,120 (  48,120)
F6     (84,)          10,164 (10,164)     10,164 (  10,164)
RBF    (10,)             840 fixed            840
total trainable 60,000; connections incl. RBF 340,908  (paper: 60,000 and 340,908)
modern re-implementation: 61,706 parameters

training image 0 (class 5), central 20x20 field  |  its fixed 7x12 output code
  ..+#########........    #######
  ..##########+.......    #######
  ..###...+++#........    ##.....
  ..##+...............    ##.....
  ..##+...............    ##.....
  .+##................    #######
  .+##................    #######
  .###...+###.........    .....##
  .###.+#####.........    .....##
  .##########.........    .....##
  .#####+..##.........    #######
  .###+....##.........    #######
  .........##.........    
  .+.......##.........    
  #####++..##.........    
  +##########.........    
  ...+#######.........    
  .......++##.........    
  ....................    
  ....................    
seed 0: test error, paper-style (nearest RBF code) 0.65%, modern (largest logit) 1.15%
seed 1: test error, paper-style (nearest RBF code) 0.65%, modern (largest logit) 1.85%
seed 2: test error, paper-style (nearest RBF code) 0.35%, modern (largest logit) 0.40%

Connections exceed parameters wherever weights are shared: C1’s 156 parameters are reused at all $28 \times 28$ positions, while in C5 and F6 each weight is used once, so the two counts coincide. The training run checks that the printed architecture trains on easy synthetic digits; it is not a comparison of the two designs, whose learning rates were set separately (0.003 for the RBF penalties, which are sums of 84 squared terms, and 0.03 for the logits).