Inductive Bias

The inductive bias of a learning system is what it assumes about the inputs it has not seen. A finite set of training images is consistent with infinitely many rules, so every learner prefers some rules to others before it sees any data. In a neural network for images, most of that preference is built into the architecture: a convolution assumes that useful patterns are small and that a pattern means the same thing wherever it appears; self-attention assumes almost nothing about where tokens sit; a graph layer assumes that the order in which nodes are listed is meaningless. These choices decide how many images a network needs and what it can learn at all, and the trade between the two runs through the history of vision architectures, from LeNet to vision transformers and back to convolution-like transformers.

Definition

An inductive bias is any assumption, other than the training data, that leads a learner to prefer one hypothesis over another that fits the data equally well. The Machine Learning article gives the definition with its sources, explains why induction from finite data needs such assumptions, and states the no free lunch theorem, which makes that necessity precise. ConViT’s authors trace the role of strong inductive biases in deep learning to Mitchell (1980) and to Goodfellow, Bengio and Courville (2016) (d’Ascoli et al., 2021, §1).

In deep learning the term is used mostly for architectural biases: constraints on which functions a layer can represent, such as locality, weight sharing, and equivariance to a group of transformations. Other choices act as biases too: the initialization, the loss, regularization, data augmentation, and pretraining on other data. This article covers the architectural biases of vision networks, the evidence on when they help, and the ways newer designs make them softer.

Intuition

Suppose an edge detector must be learned from examples. A fully connected layer has a separate weight from every input pixel to every output pixel, so it must learn the edge detector at each position separately, and it learns nothing about positions where the training images had no edges. A convolution states in advance that the output at each pixel depends only on a small neighbourhood (locality) and that the same rule applies at every pixel (weight sharing). It then has only a kernel’s worth of parameters to fit, and every position of every training image contributes evidence for the same few numbers, so one image can be enough.

The price comes when the assumption is wrong. If the correct rule differs from place to place, for example because a lens darkens the corners of every photograph, a convolution cannot represent it, however much data it sees. Goodfellow, Bengio and Courville (2016, §9.4) put both sides in Bayesian terms: a convolutional layer is a fully connected layer with an infinitely strong prior that forbids most weights, so it can make a network underfit when the assumption fails, and convolutional models should be compared with each other rather than with models that would learn as well from images whose pixels had been shuffled.

A good bias is therefore not a weak one or a strong one, but one that matches the data. Images are, to a good approximation, statistically similar at every position, locally correlated, and composed of parts, which is why the biases of convolutional networks have served vision for decades.

Formal Definition

Equivariance and invariance. Let $T_g$ be a transformation of the input indexed by an element $g$ of a group $G$ (a shift, a rotation, a permutation), and $T'_g$ its action on the output. A map $\Phi$ is equivariant to $G$ if

$$\Phi(T_g x) = T'_g \, \Phi(x) \quad \text{for all } g \in G,$$

and invariant in the special case where $T'_g$ is the identity (Cohen & Welling, 2016, §2, Eq. 1). A stride-1 convolution is equivariant to translations, up to borders; a global pooling over positions makes a feature invariant to them; self-attention without positional information is equivariant to permutations of its tokens, as the self-attention article shows.

Biases as constraints on a linear map. Every linear layer on a single-channel $H \times W$ image can be written as $y = W x$ with $W \in \mathbb{R}^{HW \times HW}$. The common layers are subsets of these matrices:

• fully connected: any $W$; $(HW)^2$ parameters;
• locally connected (“unshared convolution”, Goodfellow et al., 2016, §9.5): $W_{ij} = 0$ unless pixel $j$ lies in a $k \times k$ window around pixel $i$, with a separate kernel per output pixel; $HW k^2$ parameters;
• shared full-size kernel (circular convolution with an $H \times W$ kernel): $W_{ij} = c(i - j)$, a circulant matrix; $HW$ parameters;
• convolution: both constraints, $W_{ij} = k(i - j)$ for $i - j$ inside the $k \times k$ window and 0 otherwise; $k^2$ parameters.

For $16 \times 16$ images and $5 \times 5$ windows these are 65,536, 6,400, 256 and 25 parameters. Locality removes connections; weight sharing ties the remaining ones together; translation equivariance comes from the sharing. Goodfellow and colleagues (§9.5) note that locally connected layers suit tasks where each feature depends on a small region but there is no reason to expect the same feature everywhere, such as looking for a mouth only in the lower half of a face image. DeepFace used such layers on aligned faces.

Hard and soft biases. A constraint of this kind is a hard bias: excluded functions cannot be learned. A soft bias makes some functions easier to reach without excluding the others, through an initialization, a penalty, an auxiliary loss or augmented data. d’Ascoli et al. (2021) use the two terms in this sense.

Properties

The biases of common layers

Layer	Built-in assumptions	Equivariance	Where it is described
Fully connected	none about the input’s arrangement	none: a fixed permutation of the inputs gives an equivalent problem	Fully connected layer, multilayer perceptron
Locally connected	locality	none	Goodfellow et al. (2016, §9.5)
Convolution	locality, weight sharing over positions	translation (stride 1, up to borders)	Convolutional neural network
Pooling	small shifts should not matter	approximate invariance to small translations	Pooling
Recurrent layer	one update rule for every time step	shift in time, approximately (the state depends on where the sequence starts)	Recurrent neural network
Graph (message passing) layer	the node ordering carries no information; interactions follow the edges	permutation of nodes	Graph neural network
Self-attention	none about positions; mixing weights depend on content	permutation of tokens, until positions are added	Self-attention, positional encoding
Group convolution	convolution’s biases plus a larger symmetry group	translations and, for example, 90° rotations and flips	Cohen & Welling (2016)

Two of these generalize convolution’s weight sharing to other structures. A recurrent network applies the same update at every step; Goodfellow and colleagues (2016, Ch. 10 introduction and §10.1) explain that separate parameters for each time index could not generalize to sequence lengths not seen in training, and that a single shared model needs far fewer training examples. Message passing networks apply the same message and update functions to every node and edge, and their readout must be invariant to permutations of the node states so that the whole network is invariant to graph isomorphism (Gilmer et al., 2017, §2). Gilmer and colleagues motivate this with the symmetries of molecules, and write that the time is ripe for more flexible machine learning methods in chemistry, assuming models with suitable inductive biases can be found (§1).

Attention is the outlier. Its weights are computed from the content of each image, so a single layer can relate any two positions, but it has no built-in notion of which positions are near each other. Locality and 2D layout must come from positional encodings, masks or data.

Bias against data: approximation and estimation

The test error of a model fitted to data has two parts: the approximation error, from the best function the model class contains, and the estimation error, from fitting that function with a finite, noisy sample. A strong bias shrinks the model class. This lowers the estimation error, because fewer parameters must be pinned down by the same data, but it raises the approximation error to whatever distance separates the class from the truth. When the bias is correct, the approximation error is zero and the bias only helps. When it is wrong, more data cannot remove the error that remains.

For linear least squares the estimation error has an exact form. With design matrix $A$ (one row per equation), noise of variance $\sigma^2$ on the targets and test inputs $A_{\text{test}}$, the expected excess test error of the least-squares fit is

$$\mathbb{E}\big[\lVert A_{\text{test}}(\hat{w} - w)\rVert^2\big] = \sigma^2 \operatorname{tr}\!\big(A_{\text{test}}\,(A^\top A)^{+} A_{\text{test}}^\top\big),$$

for a $w$ in the model class, where $^{+}$ is the pseudoinverse. With fewer equations than parameters, $A^\top A$ is singular and least squares has infinitely many exact solutions. The usual choice, and NumPy’s, is the minimum-norm solution, which sets to zero every component of the weights that the training inputs never excite. That adds a bias term, the part of the true map outside the span of the training inputs, and when the number of equations is close to the number of parameters, the smallest nonzero singular values of $A$ are tiny and the variance term above becomes very large. Test error then peaks near this interpolation threshold, a form of the effect the regularization article describes as double descent. Weight sharing changes the count of equations, not only the count of parameters: one $16 \times 16$ image gives 256 equations for a shared kernel but one equation per output pixel for an unshared one.

Group equivariance

Convolution is equivariant to translations but not to other symmetries of the pixel grid. Rotating an image and then correlating it with a fixed filter equals rotating the correlation of the original image with the inversely rotated filter (Cohen & Welling, 2016, §5, Eq. 9); the authors note that if a CNN learns rotated copies of the same filter, the stack of feature maps is equivariant, although the individual maps are not. Group equivariant CNNs (G-CNNs) build the larger symmetry in. Their G-convolutions share weights across a group such as p4 (translations and rotations by multiples of 90°) or p4m (p4 plus reflections), so that feature maps transform predictably when the input is rotated or flipped (§4, §6). In the authors’ experiments, replacing each convolution by a p4-convolution, with half as many filters to keep the parameter count roughly fixed, max-pooling over rotations after the last layer and no dropout, lowered the test error on rotated MNIST from 5.03% to 2.28% (Table 1). On CIFAR-10, p4m versions of All-CNN-C and a 44-layer ResNet had lower test error than their planar counterparts with and without augmentation by flips and small translations, for example 6.46% against 9.45% for the ResNet without augmentation and 4.94% against 5.61% with it (Table 2). The authors note that CIFAR-10 is not actually symmetric, since objects usually appear upright, so a symmetry need not hold exactly for the bias to help (§9).

Examples

Weight sharing and locality, measured in closed form

The Machine Learning article traces learning curves for a linear classifier on pixels, a linear classifier on a hand-designed orientation histogram, and an MLP, on synthetic shapes; the designed features win with few images and the MLP overtakes them with many. The example below asks a narrower question with exact answers. The target is a known linear map, every model is fitted in closed form, the best error each model class can reach is known, and locality and weight sharing are separated into four models. A second target that breaks the convolution’s assumption shows a hard bias failing.

Each input is a random $16 \times 16$ grayscale crop of one of twelve images bundled with scikit-image (camera, coins, moon, page, text, brick, grass, gravel, and astronaut, coffee, chelsea and rocket converted to grayscale), with values in $[0, 1]$. The equivariant target is the crop correlated with a $5 \times 5$ derivative-of-Gaussian edge filter (horizontal derivative, $\sigma = 1$). Borders wrap around (circular cross-correlation), so the target is exactly translation equivariant on the crop. The vignetted target applies the same filter after multiplying the crop by a radial gain that falls from 1 at the centre to 0.3 at the corners, like a lens that darkens the corners of every frame: it is local but not translation equivariant. Training targets carry Gaussian noise with a standard deviation of 10% of the equivariant target’s root mean square, the same noise for both targets; the 2,000 test crops are scored against clean targets.

The four models are those of the Formal Definition: a shared $5 \times 5$ kernel (conv), a shared full-size kernel (shared 16x16), a separate $5 \times 5$ kernel per pixel (local 5x5) and an unconstrained matrix (fully connected). Each is fitted by least squares with numpy.linalg.lstsq, which returns the minimum-norm solution when the system is underdetermined. The shared full-size kernel is a circulant least-squares problem, which is diagonal in the Fourier domain: its solution is the closed form used by correlation filters, here without their ridge term, which Correlation Filter Tracking derives and checks against an explicit solve. Training sets are nested (the first $n$ of 1,024 crops) and each of seeds 0–4 draws new crops and new noise. The measure is the relative test error $\sum \lVert \hat{y} - y \rVert^2 / \sum \lVert y \rVert^2$, so 0 is perfect and 1 is the error of predicting zero. The script runs in about 15–30 seconds, depending on machine load, with NumPy 2.3.5 and scikit-image 0.25.2.

import numpy as np
from skimage import data, color

H = W = 16; D = H * W                       # 16x16 crops, 256 pixels
R = 2; KS = 2 * R + 1                       # 5x5 kernels
NAMES = ['camera', 'coins', 'moon', 'page', 'text', 'brick', 'grass', 'gravel',
         'astronaut', 'coffee', 'chelsea', 'rocket']        # all bundled with scikit-image
IMGS = [(color.rgb2gray(im) if im.ndim == 3 else im / 255.0)
        for im in (getattr(data, nm)() for nm in NAMES)]

def crops(n, rng):
    """n random 16x16 grayscale crops, values in [0, 1]."""
    out = np.empty((n, H, W))
    for i in range(n):
        im = IMGS[rng.integers(len(IMGS))]
        r, c = rng.integers(im.shape[0] - H + 1), rng.integers(im.shape[1] - W + 1)
        out[i] = im[r:r + H, c:c + W]
    return out

OFFS = [(a, b) for a in range(-R, R + 1) for b in range(-R, R + 1)]
def neigh(x):
    """(n, H, W) -> (n, D, 25): the 5x5 wrap-around neighbourhood of every pixel."""
    return np.stack([np.roll(x, (-a, -b), axis=(-2, -1)).reshape(len(x), D) for a, b in OFFS], -1)

# The unknown map: a 5x5 x-derivative-of-Gaussian edge filter (circular cross-correlation) ...
u, v = np.mgrid[-R:R + 1, -R:R + 1]
K = -v * np.exp(-(u ** 2 + v ** 2) / 2.0); K /= K[K > 0].sum()
edge = lambda x: (neigh(x) @ K.ravel()).reshape(x.shape)
# ... applied to the image itself (translation equivariant) or after a vignette (not equivariant).
ii, jj = np.mgrid[0:H, 0:W]
gain = 1 - 0.7 * ((ii - 7.5) ** 2 + (jj - 7.5) ** 2) / (2 * 7.5 ** 2)    # 1 at the centre, 0.3 at corners
TARGETS = {'equivariant': edge, 'vignetted': lambda x: edge(gain * x)}

# Four linear models, each fitted by least squares (minimum-norm solution when underdetermined).
def fit(model, X, Y):
    n = len(X)
    if model == 'conv':            # one shared 5x5 kernel: locality + weight sharing (25 parameters)
        return np.linalg.lstsq(neigh(X).reshape(-1, KS * KS), Y.reshape(-1), rcond=None)[0]
    if model == 'shared 16x16':    # one shared full-size kernel: weight sharing only (256 parameters)
        Xf, Yf = np.fft.fft2(X), np.fft.fft2(Y)      # circulant least squares is diagonal in Fourier space
        pw = (np.abs(Xf) ** 2).sum(0)
        keep = np.sqrt(pw) > np.sqrt(pw.max()) * n * D * np.finfo(float).eps   # lstsq's default cutoff
        return np.where(keep, (Yf * np.conj(Xf)).sum(0) / np.where(keep, pw, 1), 0)
    if model == 'local 5x5':       # a separate 5x5 kernel per pixel: locality only (6,400 parameters)
        nb = neigh(X)
        return np.stack([np.linalg.lstsq(nb[:, j], Y.reshape(n, D)[:, j], rcond=None)[0] for j in range(D)])
    if model == 'fully connected': # unconstrained 256x256 matrix (65,536 parameters)
        return np.linalg.lstsq(X.reshape(n, D), Y.reshape(n, D), rcond=None)[0]

def predict(model, p, X):
    if model == 'conv':            return (neigh(X) @ p).reshape(X.shape)
    if model == 'shared 16x16':    return np.real(np.fft.ifft2(p * np.fft.fft2(X)))
    if model == 'local 5x5':       return np.einsum('ndk,dk->nd', neigh(X), p).reshape(X.shape)
    if model == 'fully connected': return (X.reshape(len(X), D) @ p).reshape(X.shape)

MODELS = ['conv', 'shared 16x16', 'local 5x5', 'fully connected']
N_PARAMS = [KS * KS, D, D * KS * KS, D * D]
rel_err = lambda Yhat, Y: ((Yhat - Y) ** 2).sum() / (Y ** 2).sum()

X_test = crops(2000, np.random.default_rng(1000))
shift = lambda x: np.roll(x, (3, 5), axis=(-2, -1))
for name, T in TARGETS.items():
    gap = np.abs(T(shift(X_test[:50])) - shift(T(X_test[:50]))).max()
    print(f"{name:11s} target: max |T(shift x) - shift(T x)| = {gap:.1e}")

# Noise-free base check: with enough images, a model whose class contains the target recovers it.
X_pool = crops(1024, np.random.default_rng(0))
print("noise-free, 1,024 training images: relative test error")
for name, T in TARGETS.items():
    errs = [rel_err(predict(m, fit(m, X_pool, T(X_pool)), X_test), T(X_test)) for m in MODELS]
    print(f"  {name:11s} " + "  ".join(f"{m} {e:.3g}" for m, e in zip(MODELS, errs)))

# Learning curves: training targets carry Gaussian noise (sd = 10% of the target's rms), test targets are clean.
SIGMA = 0.1 * np.sqrt((edge(X_test) ** 2).mean())
NS = [1, 4, 16, 64, 256, 1024]
SEEDS = range(5)
err = np.zeros((len(TARGETS), len(MODELS), len(NS), len(SEEDS)))
for s in SEEDS:
    rng = np.random.default_rng(s)
    X_tr = crops(NS[-1], rng)
    noise = SIGMA * rng.standard_normal(X_tr.shape)
    for t, T in enumerate(TARGETS.values()):
        Y_tr, Y_te = T(X_tr) + noise, T(X_test)
        for m, model in enumerate(MODELS):
            for k, n in enumerate(NS):                    # nested training sets: the first n crops
                err[t, m, k, s] = rel_err(predict(model, fit(model, X_tr[:n], Y_tr[:n]), X_test), Y_te)

print(f"\nrelative test error, training noise sd {SIGMA:.4f}; seeds 0-4")
head = f"{'n':>6s}" + "".join(f"{m:>17s}" for m in MODELS)
for t, name in enumerate(TARGETS):
    print(f"{name} target, median")
    print(head)
    print(f"{'params':>6s}" + "".join(f"{p:>17,d}" for p in N_PARAMS))
    for k, n in enumerate(NS):
        print(f"{n:6d}" + "".join(f"{np.median(err[t, m, k]):17.3g}" for m in range(len(MODELS))))
    print(f"{name} target, min-max")
    for k, n in enumerate(NS):
        print(f"{n:6d}" + "".join(f"{f'{err[t, m, k].min():.2g}-{err[t, m, k].max():.2g}':>17s}" for m in range(len(MODELS))))

Output:

equivariant target: max |T(shift x) - shift(T x)| = 0.0e+00
vignetted   target: max |T(shift x) - shift(T x)| = 3.4e-01
noise-free, 1,024 training images: relative test error
  equivariant conv 7.53e-27  shared 16x16 1.45e-30  local 5x5 1.16e-28  fully connected 3.79e-27
  vignetted   conv 0.436  shared 16x16 0.435  local 5x5 4.18e-29  fully connected 2.11e-27

relative test error, training noise sd 0.0094; seeds 0-4
equivariant target, median
     n             conv     shared 16x16        local 5x5  fully connected
params               25              256            6,400           65,536
     1          0.00514            0.223             1.57             1.51
     4         0.000448          0.00861            0.514            0.926
    16         5.89e-05         0.000658             2.18            0.751
    64         1.83e-05         0.000167           0.0202            0.885
   256         4.05e-06         3.71e-05          0.00139              414
  1024         1.06e-06            1e-05         0.000282          0.00657
equivariant target, min-max
     1      0.0015-0.06        0.078-1.4            1-6.9            1-6.6
     4    9.1e-05-0.011      0.0029-0.23         0.41-1.6         0.89-1.3
    16  2.8e-05-0.00012   0.00057-0.0013         0.91-3.1          0.7-1.1
    64  8.3e-06-2.1e-05  0.00013-0.00019        0.02-0.03           0.45-1
   256    3e-06-5.4e-06  3.2e-05-4.4e-05    0.0013-0.0016  1.3e+02-1.1e+03
  1024  9.1e-07-1.6e-06  9.3e-06-1.1e-05   0.00025-0.0003    0.0059-0.0071
vignetted target, median
     n             conv     shared 16x16        local 5x5  fully connected
params               25              256            6,400           65,536
     1            0.624              2.9            0.798            0.785
     4            0.485             1.52            0.333            0.571
    16            0.442            0.475             2.34             0.53
    64            0.437            0.443           0.0217            0.876
   256            0.436            0.436          0.00149              446
  1024            0.436            0.435         0.000303          0.00708
vignetted target, min-max
     1         0.47-1.8     0.82-3.6e+02         0.59-3.5         0.61-3.1
     4         0.44-1.9          0.52-37          0.3-1.5        0.55-0.96
    16        0.44-0.47        0.45-0.58         0.97-3.3         0.49-0.9
    64        0.44-0.44        0.44-0.46      0.021-0.032           0.42-1
   256        0.44-0.44        0.44-0.44    0.0014-0.0017  1.4e+02-1.2e+03
  1024        0.44-0.44        0.43-0.43  0.00027-0.00032    0.0064-0.0077

The first lines check the set-up. The equivariant target commutes with a shift exactly, and the vignetted one does not. Without noise and with 1,024 images, every model recovers every target its class contains, to rounding error. The two shared models cannot represent the vignetted map: the best shared kernel leaves about 44% of the target’s energy unexplained, and that is their floor.

The equivariant target. Here every model’s class contains the truth, so all the error is estimation error, and the bias decides how fast it falls. From a single crop the convolution’s relative error is already 0.5% (median over five seeds; between 0.15% and 6%), because one crop supplies 256 equations for 25 unknowns. The fully connected map has 256 unknowns for each output pixel and one equation per image, so it is underdetermined until $n = 256$, and its median error stays between 0.75 and 1.5 up to $n = 64$. Above $n = 256$ all four models are in the ordinary least-squares regime, and their errors at $n = 1{,}024$ line up with their parameter counts: the full-size shared kernel, with about 10 times the convolution’s parameters, has about 9 times its error, and the locally connected model, with 256 times the parameters, about 270 times. In our reading, both ratios follow from the variance formula above, in which the error grows with the number of parameters each equation must help determine; the fully connected map, with another 10 times the parameters, has 23 times the locally connected model’s error, more than the count alone would give. In our reading, this is the variance formula still feeling the interpolation threshold: 1,024 images are four times the fully connected map’s 256 unknowns per output pixel, against about forty times the locally connected model’s 25. Correlation between the inputs cannot explain it, since least-squares predictions do not change when the inputs are transformed by an invertible linear map.

Two effects of minimum-norm least squares stand out in the printed curves. First, several errors exceed 1, worse than predicting zero. The crops are all non-negative with a large common mean, so in our reasoning any test crop has a large component along the few training crops, and the minimum-norm map answers every test crop with a scaled mixture of the training crops’ edge maps, which are wrong for the test crop and add error. Second, errors peak near the point where the number of training images reaches the number of unknowns per output pixel: the locally connected model at $n = 16$, just under its 25 unknowns per pixel (median 2.18), and the fully connected map at $n = 256$, exactly its 256 inputs per pixel (median 414, at least 130 on every seed). In our reasoning, the training design there has some very small singular values, and the noise along them is amplified, as the variance formula implies; past the threshold the error falls by orders of magnitude. The full-size shared kernel at $n = 1$ is, in our reading, the same case for weight sharing without locality: 256 equations for 256 unknowns, with a crop’s weak high frequencies giving the small singular values.

The vignetted target. The convolution and the shared kernel reach their floor of about 0.44 with 16 to 64 images and stay there: more data cannot repair a wrong assumption. The locally connected model, whose class contains the vignetted map, passes the convolution already at $n = 4$, rises above it again at its peak at $n = 16$, and from $n = 64$ on stays below it, reaching about 0.0003 at $n = 1{,}024$. The fully connected map, which also contains it, passes the convolution only at $n = 1{,}024$, after its peak at 256. Locality, the assumption that still holds, is worth keeping: from $n = 64$ on, the locally connected model’s error is more than an order of magnitude below the fully connected map’s. Weight sharing, the one that fails, is the one to drop. Which model is best therefore depends both on the data’s structure and on the number of images, the trade Goodfellow and colleagues (§9.4) describe.

The trade-off at ImageNet scale

The vision transformer drops most of the convolutional biases: as that article describes, apart from the patch extraction and the resizing of position embeddings, spatial relations between patches must be learned. Its authors’ data-scale experiments, as summarized in the Vision Transformer and ViT paper articles (the paper itself was not read for this article), found it behind comparable ResNets when trained on ImageNet alone and competitive or better after pre-training on 14 million to 300 million images. DeiT’s authors describe ViT as pre-trained with hundreds of millions of images, and ConViT’s as requiring pre-training on vast amounts of data or distillation from a convolutional network. Those two later papers addressed the small-data side.

DeiT (Touvron et al., 2021) trained a ViT-B-sized transformer on ImageNet-1k only. Without distillation it reached 81.8% ImageNet top-1 accuracy at $224 \times 224$ and 83.1% after fine-tuning at $384 \times 384$ (Table 3). The authors write that transformers need more data than models that integrate more priors, such as convolutions, and they compensated with extensive data augmentation (RandAugment and random erasing among them) and regularization such as mixup and CutMix (§6). They also distilled from a convolutional teacher through a dedicated distillation token. A convnet teacher gave better students than a transformer teacher, which they attribute, “probably”, to the inductive bias inherited through distillation, and the distilled student’s decisions were more correlated with the convnet’s than those of a transformer trained from labels only (§5.2, Tables 2 and 4).

ConViT (d’Ascoli et al., 2021) replaced some of DeiT’s self-attention layers by gated positional self-attention (GPSA). Each head starts out attending to a fixed neighbourhood of its query patch, like a convolution, and a learned gate decides how much attention it pays to position as against content (§3). On ImageNet-1k, ConViT-S reached 81.3% top-1 against DeiT-S’s 79.8%, with 27 million parameters against 22 million (Table 1). The authors measured sample efficiency by training both on a fixed fraction of each class while increasing the number of epochs so that the number of images seen stayed the same (Table 2):

Fraction of ImageNet-1k	5%	10%	30%	50%	100%
DeiT-S, top-1 (%)	34.8	48.0	66.1	74.6	79.9
ConViT-S, top-1 (%)	47.8	59.6	73.7	78.2	81.4

The gap shrinks from 13 points with 5% of the data to 1.5 points with all of it. The text gives 59.5% and 46.5% for the 10% setting, slightly different from the table, and the 100% column differs from Table 1 by 0.1 point. In ConViT-S, the average gate falls during training to zero in layers 6–10, so those layers practically ignore position, while some heads in layers 1–5 keep using it (§4). The convolutional bias is applied where it helps and dropped where it does not, which is what the authors call a soft inductive bias.

Hard and Soft Ways to Add Bias

Hybrid architectures. After ViT, many designs put convolutional biases back into transformers. The Swin Transformer restricts attention to local windows, shifted between blocks, builds a hierarchy of feature maps by merging patches, and adds a learned relative position bias; its authors note that computing attention locally has proven beneficial in modelling the high correlation in visual signals (Liu et al., 2021, §2). ConvNeXt’s authors read the success of such hierarchical transformers as the reintroduction of ConvNet priors, and asked whether the credit belonged to transformers or to those biases. Modernizing a ResNet step by step, they obtained a pure convolutional network, ConvNeXt, that performed on par with or better than similarly sized Swin Transformers, including after ImageNet-22K pre-training, against the widely held view that vision transformers, having fewer inductive biases, do better than ConvNets when pre-trained at a larger scale (Liu et al., 2022, abstract, §1, §3.1). They name translation equivariance as the most important of the ConvNets’ built-in inductive biases, desirable for tasks such as object detection (§1).

Initialization and gating. ConViT’s convolutional initialization makes local attention the starting point rather than a constraint. Tasks or layers that need global context can leave it.

Distillation. A student trained to match a teacher inherits part of the teacher’s behaviour, and DeiT’s authors attribute the advantage of a convolutional teacher to its inductive bias. In this sense knowledge distillation is a soft bias.

Data augmentation. Augmenting with shifted, flipped or rotated copies teaches the invariances that an equivariant architecture would build in, and DeiT’s recipe used augmentation to compensate for missing priors. In our reading, augmentation is the softest form of an equivariance bias: it holds only approximately, only near the transformations and images that were sampled, and costs data and compute, but it does not exclude any function. The convolutional neural network article reports a study in which augmentation taught CNNs invariance only for images similar to typical training images. The two also combine: G-CNNs still gained from augmentation by flips and small translations (Cohen & Welling, 2016, §9), and mixup and CutMix mix pairs of training images and their labels rather than encoding a symmetry.

Pretraining. Weights learned on a large dataset carry assumptions learned from other images into a small task, which is how transfer learning lets models with weak architectural biases work on small datasets.

Common Misconceptions

• “Less inductive bias is always better given enough data.” In our reading, a correct bias costs nothing at any data scale: in the example the convolution stays the most accurate model on the equivariant target at every training-set size, and ConvNeXt matched Swin after large-scale pre-training. Only a wrong bias, like weight sharing on the vignetted target, imposes a floor that data cannot remove.
• “Inductive bias means architecture.” The architecture is the most visible source, but initialization, augmentation, distillation, regularization and the optimizer’s preferences are biases too. The minimum-norm solution chosen by least squares in the example is itself a bias, toward weights that ignore whatever the training inputs never excited.
• “Equivariance and invariance are the same.” A convolution’s output moves with its input; only pooling over all positions makes it invariant. Strided convolutions and pooling break exact equivariance, as the convolutional neural network article shows.
• “Transformers have no inductive bias.” Self-attention is equivariant to permutations of its tokens, the patch embedding imposes a grid of non-overlapping patches, and the MLPs act on each token separately. In our reading its bias is weak about spatial layout, not absent.
• “Weight sharing is only about saving memory.” Sharing cuts the parameter count, but the larger effect in the example is statistical: every position of every image is evidence about the same few weights, so with one image the convolution reaches an error that the locally connected model needs 256 images to beat and the fully connected map does not reach with 1,024.

Where It Is Used

• Choosing an architecture for the data available: convolutional networks, hierarchical and windowed transformers such as Swin, and plain vision transformers sit at different points of the bias–data trade-off; ConvNeXt and ConViT test where those points lie.
• Structured inputs: recurrent networks share weights over time, graph neural networks are permutation equivariant over nodes, and point-cloud and set models are built around the same idea.
• Known symmetries: group equivariant CNNs build in rotations and reflections, as in Cohen and Welling’s experiments on rotated digits.
• Position-dependent tasks: locally connected layers on aligned inputs such as faces (DeepFace), and positional encodings that let attention models use absolute position.
• Training recipes: data augmentation, mixup and CutMix, distillation from convolutional teachers and pretraining all add soft biases, and together with the architecture they set how much labelled data a vision model needs.