Support-Vector Networks

“Support-Vector Networks” by Corinna Cortes and Vladimir Vapnik of AT&T Bell Laboratories in Holmdel, New Jersey, appeared in the journal Machine Learning in 1995 (volume 20, issue 3, pages 273–297; received May 1993, accepted February 1995). It presents a learning machine for two-class problems that maps inputs into a very high-dimensional feature space, builds a maximum-margin hyperplane there, and never forms the feature space explicitly, because it compares inputs through a function that stands in for the dot product. The paper’s own addition is the soft margin, which lets the method train on data that no hyperplane separates; it combines this with the optimal-hyperplane method and the kernel step of Boser, Guyon and Vapnik (1992), names the result the support-vector network, and tests it on handwritten digits. The method as it is understood and used today, under the name support vector machine, is covered in the support vector machine article, which owns the margin, the hinge loss, the dual, kernels and their use in vision. This article covers the paper as a publication. Equation, table and figure numbers below are those of the published journal version (the Kluwer PDF), whose notation it follows: $\ell$ training pairs $(y_i, \mathbf{x}_i)$ with labels $y_i \in \{-1, +1\}$, column vectors, and $\boldsymbol{\Lambda} = (\alpha_1, \dots, \alpha_\ell)^\top$ for the vector of Lagrange multipliers.

Problem

The paper opens with a short history of pattern recognition as the construction of decision surfaces. It credits R. A. Fisher (1936) with the first algorithm: for two Gaussian classes $N(\mathbf{m}_1, \Sigma_1)$ and $N(\mathbf{m}_2, \Sigma_2)$ the Bayes-optimal rule is quadratic (its Eq. 1), and it becomes linear when the covariances are equal (Eq. 2). Because the quadratic rule needs on the order of $n^2$ parameters, which cannot be estimated reliably from few observations (the paper’s example is fewer than $10n^2$), Fisher is said to have recommended the linear rule with a mixed covariance $\tau \Sigma_1 + (1 - \tau)\Sigma_2$ (Eq. 3) even when the covariances differ, so that pattern recognition was tied from the start to linear surfaces. Fisher’s paper as published does not say all of this; see Limitations below.

The second strand is the perceptron. In the paper’s account, Rosenblatt’s perceptron (1962) could adapt only its output weights, so its fixed hidden layers mapped inputs non-linearly into a feature space in which a linear rule was then trained (Eq. 4); back-propagation (1986, with Parker 1985 and LeCun 1985 cited alongside Rumelhart, Hinton and Williams) later let all weights adapt. The support-vector network keeps the first idea, a fixed non-linear map followed by a linear decision surface, but chooses the surface in a principled way. Figures 3 and 4 draw it as a two-layer network, to be compared with the perceptron of Figure 1: the input is compared with the stored support vectors, the results are transformed non-linearly, and a linear function of them gives the output.

The authors name two obstacles to that plan. The conceptual one is generalization: in a space of enormous dimension, many hyperplanes separate the training data, and not all of them generalize. The technical one is cost: their example is that a polynomial of degree 4 or 5 in 200 input dimensions may need a hyperplane in a billion-dimensional space. Even degree 2 in $n$ dimensions needs $n(n+3)/2$ coordinates, which they list explicitly: the $n$ inputs, their $n$ squares and the $n(n-1)/2$ cross products.

For the conceptual problem the paper points to the optimal hyperplane, which it dates to 1965, citing Vapnik’s 1982 book: the separating hyperplane with the largest margin, determined by a small subset of the training data, the support vectors. For separable data it quotes a bound (Eq. 5) on the expected test error of the optimal hyperplane,

$$\mathbb{E}[\Pr(\text{error})] \le \frac{\mathbb{E}[\text{number of support vectors}]}{\text{number of training vectors}},$$

and stresses that the dimension of the feature space does not appear in it, so a hyperplane built from few support vectors should generalize even in an infinite-dimensional space. For the technical problem it credits Boser, Guyon and Vapnik (1992) with showing that the order of operations can be swapped: compare two vectors in input space first, for instance by their dot product, and apply the non-linear transformation to the result. What was missing, according to the abstract and introduction, was a version for training data that cannot be separated without errors, since that earlier work handled only the separable case.

Contribution

The conclusion (Section 7) lists three ideas that the support-vector network combines:

1. The optimal-hyperplane solution technique, which expands the solution vector over the support vectors (Section 2).
2. Convolution of the dot product, the paper’s name for replacing $\boldsymbol{\phi}(\mathbf{u}) \cdot \boldsymbol{\phi}(\mathbf{v})$ by a function $K(\mathbf{u}, \mathbf{v})$, now called a kernel, which turns linear surfaces into non-linear ones (Section 4).
3. Soft margins, which allow errors on the training set (Section 3 and Appendix A.2).

By the paper’s own account only the third is new; the first two are presented as earlier work by Vapnik and by Boser, Guyon and Vapnik. Note 3 explains the name: “support-vector network” is meant to stress that the solution is expanded over support vectors, so that the cost of building it depends on their number and not on the dimension of the feature space. The experiments add an empirical claim: polynomial support-vector networks of degree up to 7 in 256 input dimensions do not overfit, and a degree-4 network matched the best of the specially designed neural networks in a digit-recognition benchmark.

Method

The optimal hyperplane and its dual

Section 2 reviews the separable case. A hyperplane separates the data with the canonical scaling $y_i(\mathbf{w} \cdot \mathbf{x}_i + b) \ge 1$ (Eq. 10); the distance between the projections of the two classes onto the direction $\mathbf{w}/|\mathbf{w}|$ is the margin $\rho(\mathbf{w}, b)$ (Eq. 12), which at the optimum equals $2/|\mathbf{w}_0|$ (Eq. 13), so the optimal hyperplane is the unique minimizer of $\mathbf{w} \cdot \mathbf{w}$ under those constraints, a quadratic program. Appendix A.1 derives the dual from the Lagrangian (Eqs. 42–48): $\mathbf{w}_0 = \sum_i y_i \alpha_i^0 \mathbf{x}_i$ (Eq. 14), and $\boldsymbol{\Lambda}$ maximizes

$$W(\boldsymbol{\Lambda}) = \boldsymbol{\Lambda}^\top \mathbf{1} - \tfrac{1}{2} \boldsymbol{\Lambda}^\top D \boldsymbol{\Lambda}, \qquad D_{ij} = y_i y_j\, \mathbf{x}_i \cdot \mathbf{x}_j,$$

subject to $\boldsymbol{\Lambda} \ge \mathbf{0}$ and $\boldsymbol{\Lambda}^\top \mathbf{y} = 0$ (Eqs. 15–18). The Kuhn–Tucker conditions make $\alpha_i$ non-zero only for vectors on the margin, which the appendix names support vectors. Two details are specific to the paper. First, the maximum of the dual equals $2/\rho_0^2$ (Eq. 19), so the dual doubles as a separability test: if $W$ climbs past a large constant $W_0$, no hyperplane separates the data with margin larger than $\sqrt{2/W_0}$. Second, Section 2 describes how to solve large problems by portions: solve the quadratic program on a first portion of the data, keep its support vectors, add the vectors of the next portion that violate the constraints, re-solve, and continue until all portions have been covered, with $W$ increasing monotonically along the way. Platt (1998) describes a method he attributes to Vapnik (1982), “which has since been known as chunking”, in the same terms: drop the zero multipliers, add violators, and solve a sequence of smaller quadratic programs.

The soft margin as printed

Section 3 starts from the goal of separating the data with as few errors as possible. With slack variables $\xi_i \ge 0$ and constraints $y_i(\mathbf{w} \cdot \mathbf{x}_i + b) \ge 1 - \xi_i$ (Eqs. 22–23), minimizing $\sum_i \xi_i^\sigma$ (Eq. 21) for small $\sigma > 0$ approximately counts the training errors (note 5 defines an error as a pattern with $\xi_i > 0$). Excluding those patterns and building an optimal hyperplane on the rest leads to the functional

$$\tfrac{1}{2}\mathbf{w}^2 + C\, F\Big(\sum_{i=1}^{\ell} \xi_i^\sigma\Big),$$

with $F$ monotonic and convex and $C$ a constant (Eq. 24). For large $C$ and small $\sigma$ this finds the hyperplane with the fewest training errors and the largest margin on the rest, but the authors note that minimizing the number of training errors is NP-complete in general. They therefore take $\sigma = 1$, which, for sufficiently large $C$, minimizes the sum of the deviations $\xi_i$ rather than their number while maximizing the margin of the correctly classified vectors, and which coincides with the optimal hyperplane when the data are separable. They call this solution the soft margin hyperplane, and the functional with $\sigma = 1$ is Eq. 25, with $F(0) = 0$.

The penalty is thus a function of the total slack, not a sum of per-example losses, and Section 3 works out only $F(u) = u^2$, “to simplify the formulas”. The dual then gains a scalar $\delta$: maximize $\boldsymbol{\Lambda}^\top \mathbf{1} - \tfrac{1}{2}[\boldsymbol{\Lambda}^\top D \boldsymbol{\Lambda} + \delta^2/C]$ subject to $\boldsymbol{\Lambda}^\top \mathbf{y} = 0$, $\delta \ge 0$ and $\mathbf{0} \le \boldsymbol{\Lambda} \le \delta \mathbf{1}$ (Eqs. 26–29). Since the smallest admissible $\delta$ is $\alpha_{\max} = \max_i \alpha_i$, eliminating it gives a functional with the extra term $\alpha_{\max}^2 / C$ (Eq. 30). That term, the paper says, makes the solution unique and existent for any data set; the problem is convex but no longer quadratic, so the authors solved the quadratic program in the $\ell + 1$ variables $(\boldsymbol{\Lambda}, \delta)$ instead.

Appendix A.2 generalizes. For $F(u) = u^k$ with $k > 1$ it derives the dual (Eqs. 51–66), and states that the experiments used $k = 2$; with $\sigma = 1$, the experiments’ penalty is therefore $C\big(\sum_i \xi_i\big)^2$, the square of the total slack. For $F(u) = u$ the same derivation gives the dual with box constraints $\mathbf{0} \le \boldsymbol{\Lambda} \le C\mathbf{1}$ and $\boldsymbol{\Lambda}^\top \mathbf{y} = 0$, which is the dual of the soft-margin SVM as libraries solve it today (LIBSVM’s C-SVC; Chang and Lin, 2011). The appendix then gives the dual for a general monotone convex $F$, and ends with the variant $\tfrac{1}{2}\big(\mathbf{w} \cdot \mathbf{w} + \sum_i \xi_i^2\big)$, printed without a constant $C$, whose dual adds $\boldsymbol{\Lambda}^\top \boldsymbol{\Lambda}/C$ to the quadratic form (Eq. 67); by our derivation that dual belongs to $\tfrac{1}{2}\big(\mathbf{w} \cdot \mathbf{w} + C \sum_i \xi_i^2\big)$. The paper does not report the value of $C$ used in any experiment.

Convolution of the dot product

Section 4 moves to a feature space $\boldsymbol{\phi}: \mathbb{R}^n \to \mathbb{R}^N$. Because the soft-margin solution is again an expansion $\mathbf{w} = \sum_i y_i \alpha_i \boldsymbol{\phi}(\mathbf{x}_i)$ (Eq. 32), the decision function depends on the data only through dot products $\boldsymbol{\phi}(\mathbf{x}) \cdot \boldsymbol{\phi}(\mathbf{x}_i)$ (Eq. 33), which are replaced by a symmetric function $K(\mathbf{u}, \mathbf{v})$ (Eq. 34). Hilbert–Schmidt theory expands such a $K$ in eigenfunctions (Eq. 35); $K$ is a dot product in some feature space when the eigenvalues are positive, which the paper states is guaranteed by Mercer’s condition, printed as $\iint K(\mathbf{u}, \mathbf{v})\, g(\mathbf{u})\, g(\mathbf{v})\, d\mathbf{u}\, d\mathbf{v} > 0$ for all $g$ with $\int g^2(\mathbf{u})\, d\mathbf{u} < \infty$. The authors cite Aizerman, Braverman and Rozonoer (1964), who used $K(\mathbf{u}, \mathbf{v}) = \exp(-|\mathbf{u} - \mathbf{v}|/\sigma)$ (Eq. 36) and called such functions potential functions, and give the polynomial kernel

$$K(\mathbf{u}, \mathbf{v}) = (\mathbf{u} \cdot \mathbf{v} + 1)^d$$

for a polynomial classifier of degree $d$ (Eq. 37). Training is unchanged except that the matrix of the dual becomes $D_{ij} = y_i y_j K(\mathbf{x}_i, \mathbf{x}_j)$. Section 5.2 adds that the Gaussian kernel $\exp(-|\mathbf{u} - \mathbf{v}|^2/\sigma^2)$ turns the machine into a radial basis function network that chooses its centres (the support vectors) and weights by itself, and that prior knowledge can be built into special kernels; Section 5.1 notes that the decision function is unique although its expansion over support vectors need not be (note 6). The support vector machine article explains kernels, Mercer’s condition and the kernels used in vision.

Capacity control

Section 5.3 places the method in Vapnik’s statistical learning theory. With probability $1 - \eta$, the test error is at most the training error frequency plus a confidence interval that grows with the VC dimension (Eq. 38), and structural risk minimization chooses the machine that minimizes the sum. Linear indicator functions with a fixed threshold on bounded inputs have VC dimension equal to the input dimension, but restricting the norm of $\mathbf{w}$ can make it smaller. The optimal hyperplane is therefore described as an Occam’s razor: it keeps the training error at zero and minimizes $\mathbf{w} \cdot \mathbf{w}$, which controls the second term. The parameter $C$ of the soft margin trades the two terms against each other, even when the data are separable and when no error-free solution exists.

Results

Experiments in the plane

Section 6.1 trains degree-2 polynomial machines on small two-dimensional point sets and shows the decision curves in Figure 5, with support patterns circled and errors crossed. The authors state that no degree-2 polynomial makes fewer errors on those sets and that the support patterns are few relative to the training patterns.

US Postal Service digits

The small database (Section 6.2.1) has 7,300 training and 2,000 test patterns of $16 \times 16$ pixels, recorded from actual mail. The images were centred, deslanted and smoothed with a Gaussian of $\sigma = 0.75$, the smoothing that Boser, Guyon and Vapnik (1992) had chosen. As in all the paper’s digit experiments, ten one-against-the-rest separators were trained with the same kernel and pre-processing, and a test digit was assigned to the class with the largest output (Section 6.2). Table 1 collects other classifiers’ results on this database, from publications and from experiments run for the paper, and Table 2 the support-vector networks:

Classifier (Table 1)	Raw error
Human performance (Bromley and Säckinger, 1991)	2.5%
Decision tree, CART	17%
Decision tree, C4.5	16%
Best two-layer neural network	6.6%
Special-architecture five-layer network (LeNet1)	5.1%

Degree $d$ (Table 2)	Raw error	Support vectors (mean per classifier)	Feature-space dimension (as printed)
1	12.0%	200	256
2	4.7%	127	~33,000
3	4.4%	148	~$10^6$
4	4.3%	165	~$10^9$
5	4.3%	175	~$10^{12}$
6	4.2%	185	~$10^{14}$
7	4.3%	190	~$10^{16}$

The C4.5 and two-layer-network results were obtained for the paper by Cortes and by Bernhard Schölkopf, and the CART result by Pregibon and Riley at Bell Labs. The authors read three things from Table 2. The number of support vectors grows slowly with the degree: degree 7 has about 30% more than degree 3 and fewer than degree 1, whose count of 200 includes the margin violators of a non-separable problem (34 training misclassifications per classifier on average, against 4 in total at degree 2, shown in Figure 7). The error barely changes from degree 3 to 7 while the feature space grows by ten orders of magnitude, which they read as no overfitting. And every polynomial of degree 2 or higher beats LeNet1. The training time is said to depend on the number of support vectors, not on the degree, and to be shorter even in the worst case than that of LeNet1. The bound of Eq. 5, evaluated with the observed number of support vectors in place of its expectation, held in every experiment: it never exceeded 3% for a single classifier, while the actual test error of a single classifier never exceeded 1.5%.

The last column of Table 2 is an order of magnitude. By our count, the number of distinct monomials of degree at most $d$ in 256 variables, $\binom{256 + d}{d}$, is about 33,000 at $d = 2$, as printed, but about $2.9 \times 10^6$ at $d = 3$ and $1.9 \times 10^8$, $9.7 \times 10^9$, $4.2 \times 10^{11}$ and $1.6 \times 10^{13}$ at $d = 4$ to 7, so from degree 4 on the printed values exceed that count, by a factor of about 5 at degree 4 rising to about 600 at degree 7. By the same count the degree-7 space is about $6 \times 10^6$ times the degree-3 space, not the $10^{10}$ the text reads from the table. The argument does not depend on the exact numbers.

NIST digits

The large database (Section 6.2.2) has 60,000 training and 10,000 test patterns of $28 \times 28$ pixels (784 inputs), described as a 50–50 mixture of the NIST training and test sets (note 7 names NIST Special Database 3). It was used in a benchmark study conducted over two weeks (Bottou et al., 1994), which left time for one classifier: a degree-4 polynomial on the raw pixels, without pre-processing, chosen from the US Postal Service experience. Table 3 gives each of the ten classifiers:

Classifier (Table 3)	0	1	2	3	4	5	6	7	8	9
Support patterns	1,379	989	1,958	1,900	1,224	2,024	1,527	2,064	2,332	2,765
Training errors	7	16	8	11	2	4	8	16	4	1
Test errors	19	14	35	35	36	49	32	43	48	63

The ten classifiers together misclassified 1.1% of the test set. Figure 9 compares this with the other entries of the benchmark: a linear classifier (8.4%), a 3-nearest-neighbour classifier with 60,000 prototypes (2.4%), and two networks designed for digits, LeNet1 (1.7%) and LeNet4 (1.1%). The authors note that they contributed only the support-vector results, and quote the benchmark paper’s remark that the support-vector network reached this accuracy without any knowledge of the geometry of the problem: it would do as well if the pixels were scrambled by a fixed random permutation. They conclude that kernels encoding prior knowledge should improve it further. Figure 8 shows the 14 test errors of classifier 1.

The text and Table 3 disagree on the training errors. The text gives an average training-error frequency of 0.02%, about 12 per class; Table 3 lists 77 training errors over the ten classifiers, 7.7 per classifier, or about 0.013% of 60,000. The text’s remark that the bound of Eq. 5 again holds can be checked against the table, a check the paper leaves to the reader: the support-pattern fractions run from 1.6% (classifier 1) to 4.6% (classifier 9), and the single-classifier test errors from 0.14% to 0.63%. The per-classifier test errors in the table sum to 374 binary decisions; the 1.1% figure counts digits misclassified by the combined arg-max rule, a different quantity.

The protocol on rendered shapes

The script below repeats the design of Table 2 on synthetic images with known classes: five stroke shapes (bar, cross, ring, open ring, X) rendered at $16 \times 16$ pixels with random position, rotation, length and stroke width, smoothed with the paper’s $\sigma = 0.75$ and corrupted with Gaussian noise, one image per row of the data matrix. For each degree it trains one separator per class with $K(\mathbf{u}, \mathbf{v}) = (\mathbf{u} \cdot \mathbf{v} + 1)^d$, classifies by the largest output, and checks the bound of Eq. 5 with the observed support-vector fraction, as the paper did. It uses scikit-learn’s SVC, whose penalty is $C \sum_i \xi_i$, the $F(u) = u$ case of Appendix A.2, not the squared total slack of the paper’s experiments; each image is scaled to unit norm so that $\mathbf{u} \cdot \mathbf{v} \in [-1, 1]$, a choice the paper does not specify.

import numpy as np
from scipy.ndimage import gaussian_filter
from sklearn.svm import SVC

def glyph(cls, rng, size=16):
    """Render one 16x16 image of a stroke shape with random pose, then smooth and add noise."""
    yy, xx = np.mgrid[0:size, 0:size].astype(float)
    cx, cy = size / 2 + rng.uniform(-2, 2, 2)
    th = rng.uniform(-0.35, 0.35)                     # rotation, radians
    w = rng.uniform(0.9, 1.6)                         # stroke half-width
    u = (xx - cx) * np.cos(th) + (yy - cy) * np.sin(th)
    v = -(xx - cx) * np.sin(th) + (yy - cy) * np.cos(th)
    r = np.hypot(u, v)
    L = rng.uniform(4.5, 6.5)                         # half-length / radius
    if cls == 0:   img = (abs(v) < w) & (abs(u) < L)                       # bar
    elif cls == 1: img = ((abs(v) < w) & (abs(u) < L)) | ((abs(u) < w) & (abs(v) < L))  # cross
    elif cls == 2: img = abs(r - L) < w                                     # ring
    elif cls == 3: img = (abs(r - L) < w) & (u > -1.5)                      # open ring (C shape)
    else:          img = ((abs(v - u) < 1.4 * w) | (abs(v + u) < 1.4 * w)) & (r < L)  # X
    img = gaussian_filter(img.astype(float), 0.75)    # smoothing sigma as in the paper's USPS runs
    return img + rng.normal(0, 0.25, img.shape)

def make(n_per_class, rng, k=5):
    X = np.array([glyph(c, rng).ravel() for c in range(k) for _ in range(n_per_class)])
    y = np.repeat(np.arange(k), n_per_class)
    return X / np.linalg.norm(X, axis=1, keepdims=True), y   # unit-norm rows: u.v in [-1, 1]

rng = np.random.default_rng(0)
Xtr, ytr = make(300, rng)          # 1,500 training images, one per row (n x 256)
Xte, yte = make(1000, rng)         # 5,000 test images
print(f"train {Xtr.shape}, test {Xte.shape}")
print("deg  SVs/clf  SV fraction  1-vs-rest test err  5-class test err  train errors")
for d in range(1, 8):
    scores, frac, err1, tr_err = [], [], [], 0
    for c in range(5):             # one separator per class, as in the paper
        t = np.where(ytr == c, 1, -1)
        m = SVC(kernel="poly", degree=d, gamma=1.0, coef0=1.0, C=10.0).fit(Xtr, t)  # K = (u.v + 1)^d
        frac.append(m.n_support_.sum() / len(t))                 # right-hand side of the bound (5)
        f = m.decision_function(Xte)
        err1.append(np.mean(np.sign(f) != np.where(yte == c, 1, -1)))
        tr_err += np.sum(np.sign(m.decision_function(Xtr)) != t)
        scores.append(f)
    pred = np.argmax(scores, axis=0)   # class of the maximum output
    assert all(e <= q for e, q in zip(err1, frac))               # bound holds for every separator
    print(f"{d:3d}  {np.mean(frac) * len(ytr):7.0f}  {min(frac):.3f}-{max(frac):.3f}"
          f"  {min(err1):.4f}-{max(err1):.4f}  {np.mean(pred != yte):16.4f}  {tr_err:12d}")

Output (NumPy 2.3.5, SciPy 1.16.3, scikit-learn 1.7.2):

train (1500, 256), test (5000, 256)
deg  SVs/clf  SV fraction  1-vs-rest test err  5-class test err  train errors
  1      190  0.080-0.200  0.0028-0.0914            0.0822            57
  2      254  0.111-0.249  0.0002-0.0432            0.0378             0
  3      311  0.137-0.297  0.0000-0.0322            0.0262             0
  4      362  0.169-0.333  0.0000-0.0274            0.0206             0
  5      416  0.203-0.364  0.0000-0.0242            0.0180             0
  6      488  0.242-0.409  0.0000-0.0232            0.0156             0
  7      556  0.291-0.451  0.0000-0.0230            0.0144             0

Two of the paper’s observations carry over to this data set. As in Table 2, the linear problems are not separable, and here every higher degree separates all five one-against-the-rest problems; the test error keeps falling up to degree 7 although the feature space grows by many orders of magnitude, so this run, like Table 2, shows no overfitting. The bound of Eq. 5 also holds for every separator, which the assert checks, but it is loose: the largest support-vector fraction is about twice the largest single-separator error at degree 1 and about twenty times at degree 7. The third observation does not carry over. Here the number of support vectors nearly triples from degree 1 to 7, where the paper’s grew by about 30% from degree 3 to 7. This suggests that the slow growth in Table 2 comes from the postal digits, the pre-processing, the squared-total-slack penalty and the paper’s unreported choices of $C$ and input scaling rather than from polynomial kernels in general. With a growing support set the bound weakens as the degree rises even while the actual error falls. This is a single draw of a small data set; it reproduces the protocol, not the paper’s numbers.

Impact

The Association for Computing Machinery gave Cortes and Vapnik the 2008 Paris Kanellakis Theory and Practice Award “for the development of Support Vector Machines”, which its citation calls a highly effective algorithm for classification; the citation names the 1995 paper as the one that introduced SVMs and credits it with an accurate, polynomial-time algorithm for training data that no hyperplane separates (ACM award page, archived August 2025).

The digit comparison continued after the paper. LeCun et al. (1998) describe the 60,000 training and 10,000 test images they used as the Modified NIST (MNIST) database, built from NIST Special Databases 1 and 3 (§III.A), and list Cortes and Vapnik’s 1.1% as obtained on the same data with a slightly different technique. They also report 1.4% for a regular SVM trained by Burges and Schölkopf, 1.0% for Schölkopf’s virtual support vectors and 0.8% for a modified version, and estimate the cost of the polynomial SVM at about 14 million multiply-adds per character; the reduced-set method of Burges brought the cost to 650,000 multiply-adds at 1.1% error (§III.C.11). Their best networks, boosted LeNet-4 and LeNet-5, reached 0.7% and 0.8%. The LeNet article covers that comparison.

The method’s influence on computer vision came through its use as the classifier on top of hand-designed features through the 2000s. Dalal and Triggs’s HOG detector trained a linear SVM on gradient histograms, the deformable part model trained a latent SVM, and R-CNN still used class-specific linear SVMs on CNN features in 2014. The support vector machine article surveys those uses.

Limitations

The paper states few limitations of its own. Its machine is for two classes, and the digit experiments handle ten classes by training ten separators and taking the largest output, without discussing alternatives. Only polynomial kernels were tested on real data, and on the NIST benchmark only one, of degree 4, because of the two-week schedule. The value of $C$ and the training times are not reported, and the claim that the method is faster to train than LeNet1 is given without numbers.

Later work identified cost as the main practical limit. Platt (1998) notes that the dual’s matrix has as many entries as the square of the number of training examples, which would not fit in 128 MB beyond about 4,000 examples, and that chunking reduces the matrix to roughly the square of the number of non-zero multipliers but still cannot handle large problems. At test time, LeCun et al. (1998) estimate about 14 million multiply-adds per digit for the polynomial SVM, about 35 times LeNet-5’s 401,000 (our ratio).

The paper also contains slips worth knowing when reading it:

• Fisher’s discriminant. Fisher’s 1936 paper, as published in the Annals of Eugenics (7(2), 179–188, not 7:111–132 as the reference list gives), derives the linear function of four flower measurements that maximizes the ratio of the difference between the means of two iris species to the standard deviation within species, using the pooled within-species sums of squares and products; normality enters only to estimate how often that function would misclassify a plant. We found no two-Gaussian quadratic rule like Eq. 1 in it, and no recommendation to prefer the linear rule over a quadratic one, so the paper’s account is better read as its own reconstruction than as a summary of Fisher (1936).
• Anderson and Bahadur. Cited as 1966 for the optimal $\tau$ in Eq. 3 (note 1) and for “general forms of the dot-product in a Hilbert space” (Section 4), the paper in the reference list, on classification into two multivariate normal distributions with different covariance matrices, appeared in 1962 (Annals of Mathematical Statistics, 33(2)).
• Internal inconsistencies (our reading of the printed text). The introduction twice points to Section 5 for the experiments, which are in Section 6, and Section 5.1 points to Section 3 for the portion-by-portion solution technique described in Section 2. The Gaussian decision function of Section 5.2 is printed without the minus sign in the exponent that its kernel has. At $k = 2$, Appendix A.2’s dual (Eq. 60) penalizes $\delta^2/(4C)$, by our evaluation, where Section 3’s Eq. 26 has $\delta^2/(2C)$; the two differ by a rescaling of $C$, and our own derivation from the primal $\tfrac{1}{2}\mathbf{w} \cdot \mathbf{w} + C\big(\sum_i \xi_i\big)^2$ gives the appendix’s $\delta^2/(4C)$. Appendix A.2 also says “maximize” for the primal functional and “minimize” for the dual of $F(u) = u$, the reverse of what the derivation does. The NIST training-error count in the text differs from Table 3, as noted above.

What Came After

Two sources date the method differently. Cortes and Vapnik place the optimal hyperplane in 1965; Platt (1998) writes that Vapnik invented support vector machines in 1979. Both cite Vapnik’s 1982 book, and Platt credits Cortes and Vapnik (1995) with the modification that allows and penalizes margin violations.

Training algorithms replaced chunking within a few years. Platt describes a 1997 theorem by Osuna and colleagues that justified solving fixed-size sub-problems, and his own sequential minimal optimization (SMO), which at every step solves the smallest possible sub-problem, two multipliers, analytically; his report already uses the name support vector machines. LIBSVM (Chang and Lin, 2011) states its C-SVC as exactly the $F(u) = u$ problem of Appendix A.2, crediting Boser, Guyon and Vapnik (1992) and Cortes and Vapnik (1995), so the variant the paper derived but did not run became the standard one, rather than the squared total slack of its experiments. The support vector machine article covers the method as it is used today, and the k-nearest neighbors article another classifier from the same benchmark.