Invariance and Equivariance

A feature is invariant to a transformation of the image when its value does not change, and equivariant when it changes in a way fixed in advance by the transformation: shift the image and a convolutional feature map shifts with it; rotate a patch and a detector’s orientation estimate turns by the same angle. Feature detectors and descriptors use the pair in the form the image features article describes, with covariant detectors (the classical literature’s word for equivariant ones) and invariant descriptors computed in the frame the detector reports. Neural networks build the same properties into their layers: a convolutional network is equivariant to translations because it shares weights over positions, and group-equivariant networks extend the sharing to rotations and reflections. This article is the site’s hub for the two properties. It gives the definitions, the four ways of obtaining them (sharing weights over a group, pooling over a group, canonicalizing, and normalizing), the reason most of them hold exactly on a pixel grid only for some transformations, and the trade they all make: what a representation is invariant to, it can no longer report.

Definition

Let a set of transformations $G$ act on images (shifts, rotations, rescalings, changes of illumination), with $T_g$ the action of $g \in G$ on an image $x$. A map $\Phi$ from images to features is equivariant to $G$ if, for every $g$, there is a transformation $T'_g$ of the features such that transforming the input and then computing the features gives the same result as computing the features and then transforming them. It is invariant when $T'_g$ is the identity for every $g$, so the features ignore the transformation altogether. Invariance is therefore a special case of equivariance (Cohen & Welling, 2016, §2; Lenc & Vedaldi, 2015, §2).

The words have a history of their own. Feature-detection papers call a detector covariant when its outputs move with the image; Lenc and Vedaldi (2015) write “equivariant (also called co-variant) detectors”. Signal processing calls a filter shift-invariant when it commutes with shifts, which is equivariance in the vocabulary used here: the convolution article uses the signal-processing sense when it says that every linear, shift-invariant operation is a convolution.

Intuition

Think of a corner detector run on a photograph and on the same photograph turned by 30°. A useful detector finds the same corner in both, at the turned position, and reports an orientation 30° larger. Its output has changed, but in a way one can predict from the transformation alone, which is what makes it possible to match the two views. Now take a descriptor of the patch around the corner computed after turning the patch back by the reported orientation. It should be the same in both images: it has been made invariant by using the equivariant detector’s output to undo the rotation.

The same picture explains why equivariance is often the more useful property inside a system. An invariant map throws away the transformation; an equivariant one keeps it, in a known form. Cohen and Welling (2016, §2) argue that inside a deep network equivariance is more useful than invariance, because invariant features cannot tell whether parts are in the right spatial configuration. Worrall et al. (2017, §1) make the same point with a butterfly detector: a detector invariant to the pose of the wings would accept a butterfly with impossibly rotated wings, while an equivariant one passes the pose on to later stages. Invariance is usually wanted only at the end, for a class label that should not depend on where or at what angle the object appears.

Formal Definition

A group of transformations is a set closed under composition and inverses that contains the identity. Translations of the pixel grid $\mathbb{Z}^2$ form a group; so do the four rotations by multiples of 90° about a pixel; their combination, all translations and quarter turns about any centre of the square grid, is the group Cohen and Welling (2016, §4.2) call p4, and adding reflections gives p4m. A transformation that maps an object onto itself is a symmetry of that object; flipping the grid over is their example (§4.1). Dieleman et al. (2016, §2) note that the quarter turns are the only rotations that map the square grid onto itself.

With $T_g$ the action of $g$ on inputs and $T'_g$ an action on outputs, $\Phi$ is equivariant to $G$ if

$$\Phi(T_g x) = T'_g \, \Phi(x) \quad \text{for all } g \in G \text{ and all } x,$$

and invariant if $T'_g$ is the identity (Cohen & Welling, 2016, Eq. 1). The output action need not equal the input action: Cohen and Welling require only that $T'$ composes like the group, $T'_{gh} = T'_g T'_h$, and Worrall et al. (2017, §3.1) stress that the feature-space transformation need not look like a rotation at all. Dieleman, De Fauw and Kavukcuoglu (2016, §3) name the case $T'_g = T_g$ same-equivariance: the output transforms exactly as the input does, as, for instance, a segmentation map should when the image is rotated.

Lenc and Vedaldi (2015, Eq. 1) use a looser, empirical form for representations that were not designed to be equivariant: $\Phi$ is equivariant to $g$ if there is a map $M_g$ with $\Phi(g x) \approx M_g \Phi(x)$ for all images. Since an invertible $\Phi$ always admits such a map, they ask that $M_g$ be simple, preferably linear. For HOG and a horizontal flip, $M_g$ is a permutation: it swaps cells left to right and, inside each cell, the bins of mirrored gradient orientations (their §2 example; see histogram of oriented gradients).

Equivariant layers

A correlation $[f \star \psi](x) = \sum_y f(y)\, \psi(y - x)$ commutes with shifts: shifting $f$ by $t$ shifts the output by $t$ (Cohen & Welling, 2016, Eq. 8). It does not commute with rotations. For a rotation $r$ of the grid,

$$[L_r f] \star \psi = L_r\big[f \star L_{r^{-1}} \psi\big],$$

so correlating a rotated image with a filter is the rotation of the original image correlated with the inversely rotated filter (Eq. 9). If a layer contains every rotated copy of each filter, a rotation of the input permutes its output maps among the copies as well as rotating them. The first-layer G-correlation of a G-CNN (the lifting layer, in Kaba et al.’s term) does exactly this: its output is a function on the group,

$$[f \star \psi](g) = \sum_{y \in \mathbb{Z}^2} \sum_k f_k(y)\, \psi_k(g^{-1} y),$$

one map per filter and per rotation (Eq. 10), and later layers correlate functions on the group with filters on the group, $[f \star \psi](g) = \sum_{h \in G} \sum_k f_k(h)\, \psi_k(g^{-1} h)$ (Eq. 11). Both are equivariant by the same substitution that proves Eq. 8 (Eq. 12). Pointwise nonlinearities commute with the group action (§6.2), and so does pooling over a neighbourhood performed without stride (§6.3). Pooling over a whole subgroup at each position, such as the maximum over the four rotations of a p4 map, gives a map that transforms like the input image (§6.3).

Kondor and Trivedi (2018) proved that this structure is not only sufficient but necessary. For feed-forward networks whose layers are functions on quotient spaces of a compact group, a network is equivariant to the group exactly when each of its linear layers is a generalized convolution over that group (their Theorem 1).

Invariance from equivariance

Two general routes turn an equivariant computation into an invariant one.

• Pooling over the group. Combining the responses to all transformed versions with a symmetric function, such as the maximum or the mean over the four quarter turns, gives a value that does not depend on which of them the input was. Cohen and Welling (2016, §3) list group averaging, together with pose normalization, as the ways invariance had been obtained; Dieleman et al. (2016, §4.2) pool predictions over four rotated copies of each input.
• Canonicalization. An equivariant estimator $h$ reports the input’s pose, and the input is transformed back by $h(x)^{-1}$ before an arbitrary function $f$ is applied. Kaba et al. (2023, §3.2) write the result as $\Phi(x) = h'(x)\, f\big(h(x)^{-1} x\big)$ and show that it is equivariant for any $f$ as long as $h$ is equivariant, $h(T_g x) = T_g h(x)$; with $h'$ the identity it is invariant. A keypoint descriptor computed in the frame of the detected orientation and scale is the classical instance: Lowe (2004, §5) assigned each SIFT keypoint an orientation so that the descriptor, represented relative to it, is invariant to image rotation.

Canonicalization has a weak point at inputs that are themselves symmetric. If a patch looks the same after a rotation by 180°, an equivariant estimator would have to report two poses at once. Kaba et al. (2023, Appendix A) show that no canonicalization function satisfies the standard definition at such inputs and introduce a relaxed form for them. Lowe (2004, §5) handled the practical version of the problem by creating an extra keypoint for every orientation-histogram peak within 80% of the highest, and reported that only about 15% of keypoints received more than one orientation.

Properties

Composition. Equivariant maps compose: if $\Phi_1(T_g x) = T'_g \Phi_1(x)$ and $\Phi_2(T'_g y) = T''_g \Phi_2(y)$, then $\Phi_2 \circ \Phi_1$ is equivariant with output action $T''_g$ (our derivation from the definition). An equivariant stack followed by one invariant step is invariant. One non-equivariant layer anywhere breaks the chain: a strided convolution or a pooling layer is equivariant only to shifts by multiples of its stride, which is why, as the pooling article shows, a one-pixel shift can change a CNN’s output; the sampling and aliasing article traces this to subsampling without a low-pass filter.

Exact only for the grid’s symmetries. On a pixel grid, a transformation can be applied without resampling only if it maps pixel centres onto pixel centres: integer shifts, quarter turns, and reflections. Dieleman et al. (2016, §2) chose cyclic symmetry for this reason, noting that only the four rotations by multiples of 90° allow a filter to be applied without interpolation. Every other rotation and every rescaling resamples the image, and the equivariance that holds for continuous images holds only approximately on pixels. Harmonic networks (Worrall et al., 2017) restrict filters to circular harmonics $W_m(r, \phi) = R(r)\, e^{i(m\phi + \beta)}$, whose responses to a rotated patch differ only by a phase factor $e^{im\theta}$ (§3.2), giving equivariance to all rotations in the continuum; on pixels they band-limit the input with a Gaussian blur (§4.2), and their measured response magnitude under input rotation shows small periodic fluctuations, which they attribute to the inexactness of the anti-aliasing (§5.2). The same limit appears in classical features: Hu’s moment invariants are exact for continuous shapes and drift on pixels, as the image moments article measures, and rotation-invariant local binary patterns are exact for quarter turns and only approximate at other angles.

Each invariance discards information. An invariant map gives the same output for every member of an orbit, so it cannot tell them apart. The image features article sets out the trade for descriptors, and representation learning gives the learned version: a representation invariant to orientation cannot report orientation. Equivariance avoids the loss, which is one reason networks keep it until the last layers.

Equivariance is relative to a group, and so are the guarantees. A G-CNN for p4 says nothing about rotations by 45°, and a colour normalization exact under one model of illumination change says nothing about another. Cohen and Welling (2016, §9) note that G-CNNs still benefit from data augmentation when it comes from a group larger than the one built in, such as flips and small translations for p4 and p4m networks.

Approximate symmetry can still help. Cohen and Welling (2016, §9) observe that CIFAR-10 is not rotation-symmetric, since objects usually appear upright, yet p4m networks lowered the error there; the inductive bias article gives their numbers. On rotated MNIST, Worrall et al. (2017, Table 1) reported 1.69% test error for their harmonic network against 2.28% for Cohen and Welling’s P4CNN and 5.03% for a plain CNN, with 33k, 25k and 22k parameters in the table (their text gives 22k for Cohen and Welling’s network as well).

Kinds of Invariance in Vision

Translation. Convolution gives translation equivariance and global pooling gives invariance; the convolutional neural network article tests the equivariance and measures how a network with global pooling fares on shifted digits, and the Pooling article shows how subsampling weakens them.

Rotation. Three families coexist. Hand-crafted detectors estimate an orientation and descriptors canonicalize with it: SIFT uses the peak of a gradient-orientation histogram (Lowe, 2004, §5) and ORB the direction to the intensity centroid. The feature descriptors example shows that rotating the descriptor grid to the dominant orientation rescues a rotated patch but makes the descriptor move more under noise, and ORB’s example measures matches over rotations from 0° to 180° with and without steering. Rotation-invariant pattern codes, such as the rotation-invariant mappings of local binary patterns, give one label to all cyclic shifts of a code. Networks either share filters over a rotation group (p4 G-CNNs; cyclic slicing, pooling, rolling and stacking in Dieleman et al., 2016), restrict them to steerable families (harmonic networks), or learn a canonicalization function in front of an ordinary network (Kaba et al., 2023).

Scale. A rescaled image is not a subset of the original grid, so scale is handled by representing the image at many scales. In a Gaussian scale space, γ-normalized derivative responses take their maxima at scales that move with the image (covariance), and their values at the maxima are equal across rescalings only for $\gamma = 1$, as that article derives from Lindeberg’s work; the image pyramid is the sampled form. Detectors such as SIFT report the selected scale, and descriptors are computed on a patch of that size.

Affine and viewpoint. Affine-covariant detectors adapt an elliptical region to the local second-moment matrix, as described in feature detection, so that, as Scale Space describes it, the normalized region is determined up to a scale factor and a rotation, which scale selection and orientation assignment remove.

Illumination. Gradients ignore an additive offset and normalization removes a gain, which together make the SIFT descriptor invariant to affine intensity changes, as Image Features summarizes; local binary patterns whose neighbours lie on pixel centres are invariant to every strictly increasing grey-level map, while interpolated neighbours keep the invariance exactly only for gain and offset, as that article shows. For colour, van de Sande, Gevers and Snoek (2010, §II) start from the diagonal model of illumination change, in which each channel is multiplied by its own factor, $(R, G, B) \mapsto (aR, bG, cB)$, and, following an extension they attribute to Finlayson and colleagues, add an offset per channel (the diagonal-offset model). They name five special cases: light intensity change ($a = b = c$), light intensity shift (equal offsets), the two combined, light colour change ($a \ne b \ne c$), and light colour change and shift, and tabulate which descriptors are invariant to each (their Table I). Their analysis assumes that no colour is clipped. Global image descriptors applies the table to colour histograms and measures which descriptor survives which change in retrieval; the Examples below check the table itself numerically, pixel by pixel.

Learned invariances. Networks trained without built-in symmetry become approximately invariant to what their training data and augmentations vary. Lenc and Vedaldi (2015, §3.2) learned linear maps $M_g$ for the layers of an ImageNet-trained AlexNet and found that for horizontal flips and rescaling, transformations the network had become invariant to, the learned maps did no better than leaving the features unchanged, with invariance obtained largely in the third or fourth convolutional layer; for vertical flips and 90° rotations, transformations the network had not been trained to ignore, the maps reduced the error substantially and far fewer feature channels were invariant. Dieleman et al. (2016, §4) note that even a network that learns an invariance perfectly on its training set has no guarantee that it generalizes.

Examples

The examples test three of these mechanisms on bundled scikit-image images, with NumPy 2.3.5, SciPy 1.16.3 and scikit-image 0.25.2. The whole script runs in about 7 seconds on one CPU thread, and two runs printed identical output. The hypotheses and tolerances below were written down before the script was first run.

Conventions: arrays are indexed (row, column) with rows pointing down; rotations are counter-clockwise as displayed, about the centre pixel of a $129 \times 129$ crop; np.rot90 performs the quarter turns, and scipy.ndimage.rotate the other angles. Orientations are measured counter-clockwise from the $+x$ axis with $y$ pointing up. Correlations use zero padding, as in deep-learning libraries’ “convolutions”.

A p4 group convolution is exactly equivariant

The first part builds the layers of the Formal Definition on a crop of camera. The filters are random: eight $9 \times 9$ first-layer filters and $4 \times 8 \times 4$ second-layer filters, each a Gaussian-smoothed random array masked to a disk of radius 4 and scaled to unit norm, standing in for learned filters. The check compares the two sides of each identity after a quarter turn of the input, as a relative error $\lVert a - b \rVert / \lVert b \rVert$. A rotation acts on p4 feature maps by rotating each map and shifting the rotation index by one (rot_p4), the action derived from Eq. 10 and 11.

# ---- Part A: exact equivariance of a p4 group convolution ----
import os
for v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS"):
    os.environ[v] = "1"
import numpy as np
from scipy import ndimage
from skimage import data, color

N, R_DISK, KS = 129, 40, 9                 # crop size, evaluation disk radius, filter size
yy, xx = np.mgrid[:N, :N] - N // 2
DISK = yy**2 + xx**2 <= R_DISK**2

def gray_crop(name):
    a = getattr(data, name)()
    a = color.rgb2gray(a) if a.ndim == 3 else a / 255.0
    i, j = (a.shape[0] - N) // 2, (a.shape[1] - N) // 2
    return a[i:i + N, j:j + N].astype(np.float64)

def filter_bank(k, seed):
    """Smooth random filters on a disk of radius 4, unit norm (stand-ins for learned filters)."""
    rng = np.random.default_rng(seed)
    c = np.mgrid[:KS, :KS] - KS // 2
    mask = (c[0]**2 + c[1]**2) <= (KS // 2) ** 2
    w = np.array([ndimage.gaussian_filter(rng.standard_normal((KS, KS)), 1.0, mode="constant") * mask
                  for _ in range(k)])
    return w / np.linalg.norm(w.reshape(k, -1), axis=1)[:, None, None]

def corr(x, w):                            # cross-correlation, zero padding
    return ndimage.correlate(x, w, mode="constant", cval=0.0)

def rel(a, b):
    return np.linalg.norm(a - b) / np.linalg.norm(b)

relu = lambda z: np.maximum(z, 0.0)
rot = lambda a, k=1: np.rot90(a, k, axes=(-2, -1))   # 90 degrees counter-clockwise

def lift(x, W):
    """Lifting layer: image -> (K, 4, H, W), one map per filter and rotation r."""
    return np.array([[corr(x, rot(w, r)) for r in range(4)] for w in W])

def gconv(f, W2):
    """p4 -> p4 group correlation: out[o, r] = sum_{c,s} f[c, s] * rot^r(W2[o, c, s - r])."""
    O, C = W2.shape[:2]
    out = np.zeros((O, 4) + f.shape[2:])
    for o in range(O):
        for r in range(4):
            for c in range(C):
                for s in range(4):
                    out[o, r] += corr(f[c, s], rot(W2[o, c, (s - r) % 4], r))
    return out

def rot_p4(F):
    """How a 90-degree rotation acts on p4 feature maps: rotate each map, shift r by one."""
    return rot(np.roll(F, 1, axis=1))

W1 = filter_bank(8, seed=0)
W2 = filter_bank(4 * 8 * 4, seed=1).reshape(4, 8, 4, KS, KS)
x = gray_crop("camera")
xr = rot(x)

print("Part A: 90-degree rotation of a 129 x 129 crop of 'camera' (relative errors)")
print(f"  plain correlation, rot(x) * w vs rot(x * w), median over 8 filters "
      f"{np.median([rel(corr(xr, w), rot(corr(x, w))) for w in W1]):.2e}")
print(f"  rot(x) * w vs rot(x * rot^-1(w)), largest over 8 filters        "
      f"{max(rel(corr(xr, w), rot(corr(x, rot(w, -1)))) for w in W1):.2e}")
F1, F1r = lift(x, W1), lift(xr, W1)
print(f"  lifting layer:       F(rot x) vs rot_p4(F(x))                    {rel(F1r, rot_p4(F1)):.2e}")
G, Gr = relu(gconv(relu(F1), W2)), relu(gconv(relu(F1r), W2))
print(f"  + ReLU, p4 layer, ReLU: G(rot x) vs rot_p4(G(x))                 {rel(Gr, rot_p4(G)):.2e}")
pool = lambda H: H.max(axis=1)[:, DISK].mean(axis=1)    # max over rotations, mean over disk
print(f"  max over r, mean over disk: invariance error                     {rel(pool(Gr), pool(G)):.2e}")
plain2 = lambda z: relu(np.array([sum(corr(relu(corr(z, W1[c])), W2[o, c, 0]) for c in range(8))
                                  for o in range(4)]))[:, DISK].mean(axis=1)
print(f"  same depth without the group structure: invariance error         {rel(plain2(xr), plain2(x)):.2e}")

Output:

Part A: 90-degree rotation of a 129 x 129 crop of 'camera' (relative errors)
  plain correlation, rot(x) * w vs rot(x * w), median over 8 filters 3.25e-01
  rot(x) * w vs rot(x * rot^-1(w)), largest over 8 filters        3.48e-16
  lifting layer:       F(rot x) vs rot_p4(F(x))                    2.80e-16
  + ReLU, p4 layer, ReLU: G(rot x) vs rot_p4(G(x))                 1.19e-15
  max over r, mean over disk: invariance error                     7.02e-16
  same depth without the group structure: invariance error         3.51e-01

We expected the plain correlation to fail the check (relative error of at least 0.1) and every group-structured line to pass it to within $10^{-12}$; all six outcomes held. A plain correlation of the rotated crop differs from the rotated correlation by about a third of its norm (0.325), because the random filters have no rotational symmetry. Eq. 9 itself holds to rounding error (3.5e-16), and so do the lifting layer (2.8e-16), the p4 layer with its two ReLUs (1.2e-15), and the invariant descriptor made by taking the maximum over the four rotations and averaging over a disk (7.0e-16). A network of the same depth and filters without the group structure changes by 0.351. The exactness rests on the grid: a quarter turn of a square crop maps pixels onto pixels and the zero padding onto itself, so every sum is the same sum in another order.

Between the group elements: pooling against canonicalization

The second part asks what happens at angles the group does not contain. It makes one feature function rotation-invariant in four ways and measures every way with the same error. The feature is, for each of the eight first-layer filters, the ReLU response, its maximum over the filter’s rotated copies, and the mean over a disk of radius 40 about the centre. The methods are:

• no pooling: one copy per filter, no mechanism;
• pool C4: maximum over the four quarter-turned copies, exact for quarter turns;
• pool C8: eight copies, adding the four quarter turns of a 45° copy made by interpolating the filter;
• canon. estimated: rotate the crop back by its dominant gradient orientation, estimated as SIFT does in outline (Lowe, 2004, §5: a 36-bin histogram of gradient orientations weighted by magnitude and a Gaussian window, here of standard deviation 20 pixels, with a parabola through the highest bin and its neighbours; only the highest peak is used), then apply the unpooled features;
• canon. true angle: rotate the crop back by the angle it was actually rotated by, which isolates the cost of resampling.

Each of six crops (camera, astronaut, coffee, chelsea, rocket, coins, in grey) is rotated by 0° to 90° in steps of 15°. Interpolation is a modelling choice the sources leave open, so the code prints both bilinear (order 1) and cubic-spline (order 3) interpolation, used for the image rotations and for the 45° filter copies alike. The error is the relative change of the eight-number feature vector; for estimated canonicalization it is measured against the canonicalized unrotated crop. The last row of each table gives the orientation estimate’s error, $\lvert \hat\theta(R_\alpha x) - \hat\theta(x) - \alpha \rvert$, and the data line gives, for each crop, the height of the second-highest histogram peak relative to the highest.

# ---- Part B: rotations between the group elements; pooling against canonicalization ----
NAMES = ["camera", "astronaut", "coffee", "chelsea", "rocket", "coins"]
ANGLES = [0, 15, 30, 45, 60, 75, 90]

def rotate(img, deg, order):               # counter-clockwise about the crop centre
    return ndimage.rotate(img, deg, reshape=False, order=order, mode="reflect")

def bank_copies(W, n, order):
    """Rotated copies of each filter: n = 1 (none), 4 (exact quarter turns) or 8 (adds 45-degree copies)."""
    if n == 1:
        return W[:, None]
    base = [W] if n == 4 else [W, np.array([rotate(w, 45, order) for w in W])]
    return np.stack([rot(b, r) for r in range(4) for b in base], axis=1)

def describe(img, copies):
    """For each filter: ReLU response, max over its rotated copies, mean over the disk."""
    return np.array([np.max([relu(corr(img, w)) for w in ws], axis=0)[DISK].mean() for ws in copies])

def orientation(img, sig_s=2.0, sig_w=20.0):
    """Dominant gradient orientation (degrees, counter-clockwise), 36-bin histogram, parabola."""
    L = ndimage.gaussian_filter(img, sig_s)
    gy, gx = np.gradient(L)
    th = np.arctan2(-gy, gx)                # rows point down; flip to y up
    w = np.hypot(gx, gy) * np.exp(-(xx**2 + yy**2) / (2 * sig_w**2))
    h, _ = np.histogram(th[DISK], bins=36, range=(-np.pi, np.pi), weights=w[DISK])
    k = int(np.argmax(h)); l, r = h[k - 1], h[(k + 1) % 36]
    peaks = [h[i] for i in range(36) if h[i] > h[i - 1] and h[i] >= h[(i + 1) % 36] and i != k]
    off = 0.5 * (l - r) / (l - 2 * h[k] + r)
    return np.degrees(-np.pi + (k + 0.5 + off) * 2 * np.pi / 36), max(peaks, default=0) / h[k]

wrap = lambda a: (a + 180.0) % 360.0 - 180.0
crops = [gray_crop(n) for n in NAMES]
print("\nPart B data: second orientation peak / highest peak per crop:",
      " ".join(f"{n} {orientation(c)[1]:.2f}" for n, c in zip(NAMES, crops)))
METHODS = ["no pooling", "pool C4", "pool C8", "canon. estimated", "canon. true angle"]
for order in (1, 3):
    banks = {1: bank_copies(W1, 1, order), 4: bank_copies(W1, 4, order), 8: bank_copies(W1, 8, order)}
    err = {m: np.zeros((len(crops), len(ANGLES))) for m in METHODS}
    oerr = np.zeros((len(crops), len(ANGLES)))
    for i, c in enumerate(crops):
        th0 = orientation(c)[0]
        ref = {n: describe(c, banks[n]) for n in (1, 4, 8)}
        ref_canon = describe(rotate(c, -th0, order), banks[1])
        for j, a in enumerate(ANGLES):
            ca = rotate(c, a, order)
            for m, n in zip(METHODS[:3], (1, 4, 8)):
                err[m][i, j] = rel(describe(ca, banks[n]), ref[n])
            tha = orientation(ca)[0]
            oerr[i, j] = abs(wrap(tha - th0 - a))
            err["canon. estimated"][i, j] = rel(describe(rotate(ca, -tha, order), banks[1]), ref_canon)
            err["canon. true angle"][i, j] = rel(describe(rotate(ca, -a, order), banks[1]), ref[1])
    print(f"\nPart B, interpolation order {order}: invariance error, median over 6 crops [max]")
    print(f"{'rotation (deg)':18s}" + "".join(f"{a:>15d}" for a in ANGLES))
    for m in METHODS:
        print(f"{m:18s}" + "".join(f"{np.median(e):>8.1e} [{e.max():.0e}]".rjust(15) for e in err[m].T))
    print(f"{'orient. error deg':18s}" + "".join(f"{np.median(e):>7.2f} [{e.max():.1f}]".rjust(15) for e in oerr.T))

Output:

Part B data: second orientation peak / highest peak per crop: camera 0.97 astronaut 0.94 coffee 0.34 chelsea 0.16 rocket 0.91 coins 0.99

Part B, interpolation order 1: invariance error, median over 6 crops [max]
rotation (deg)                  0             15             30             45             60             75             90
no pooling         0.0e+00 [0e+00] 3.4e-03 [8e-03] 4.2e-03 [2e-02] 5.3e-03 [3e-02] 7.2e-03 [4e-02] 9.3e-03 [5e-02] 1.1e-02 [6e-02]
pool C4            0.0e+00 [0e+00] 6.4e-03 [1e-02] 9.9e-03 [2e-02] 9.3e-03 [2e-02] 6.4e-03 [1e-02] 5.2e-03 [1e-02] 1.1e-18 [1e-16]
pool C8            0.0e+00 [0e+00] 5.0e-03 [2e-02] 7.4e-03 [1e-02] 8.1e-03 [1e-02] 6.4e-03 [1e-02] 4.0e-03 [1e-02] 1.0e-16 [1e-16]
canon. estimated   0.0e+00 [0e+00] 1.5e-03 [1e-02] 1.1e-03 [1e-02] 8.2e-04 [1e-02] 4.4e-04 [1e-02] 1.7e-03 [1e-02] 1.1e-16 [3e-16]
canon. true angle  0.0e+00 [0e+00] 9.7e-04 [4e-03] 9.7e-04 [4e-03] 1.1e-03 [4e-03] 9.8e-04 [4e-03] 9.8e-04 [4e-03] 0.0e+00 [0e+00]
orient. error deg      0.00 [0.0]    3.39 [82.6]    0.47 [83.5]   3.29 [177.5]   0.39 [178.5]   3.50 [177.7]     0.00 [0.0]

Part B, interpolation order 3: invariance error, median over 6 crops [max]
rotation (deg)                  0             15             30             45             60             75             90
no pooling         2.7e-16 [3e-16] 2.9e-03 [8e-03] 4.2e-03 [2e-02] 6.2e-03 [3e-02] 8.0e-03 [4e-02] 9.7e-03 [5e-02] 1.1e-02 [6e-02]
pool C4            2.9e-16 [3e-16] 6.1e-03 [1e-02] 1.2e-02 [2e-02] 1.2e-02 [2e-02] 7.3e-03 [1e-02] 2.8e-03 [8e-03] 2.9e-16 [4e-16]
pool C8            2.6e-16 [3e-16] 3.1e-03 [5e-03] 3.9e-03 [8e-03] 5.7e-03 [9e-03] 5.7e-03 [7e-03] 3.4e-03 [5e-03] 2.5e-16 [3e-16]
canon. estimated   2.5e-16 [3e-16] 6.1e-04 [1e-02] 1.9e-04 [1e-02] 5.0e-04 [3e-03] 9.8e-05 [1e-02] 1.9e-03 [1e-02] 2.7e-16 [4e-16]
canon. true angle  5.0e-16 [7e-16] 6.2e-05 [2e-04] 6.1e-05 [2e-04] 9.5e-05 [2e-04] 5.9e-05 [2e-04] 5.8e-05 [2e-04] 5.0e-16 [7e-16]
orient. error deg      0.00 [0.0]    3.32 [82.6]    0.60 [83.5]     3.02 [3.8]   0.40 [178.7]   3.51 [177.7]     0.00 [0.0]

Before the run we fixed six expectations. Their outcomes, under each interpolation order:

• Exact at 90° (maximum over crops at most $10^{-12}$ for pool C4, pool C8 and true-angle canonicalization, at most $10^{-9}$ for estimated canonicalization): held under both orders. The maxima are 1e-16, 1e-16, 0 and 3e-16 with order 1, and 4e-16, 3e-16, 7e-16 and 4e-16 with order 3. The C8 bank is exact at 90° because its eight copies are closed under quarter turns, even though the 45° copies were interpolated; the orientation estimate moves by exactly 90° under a quarter turn, so canonicalization lands on the same grid.
• HB1, “no pooling” has the largest median error at every angle from 15° to 75°: failed under both orders, at 15°, 30° and 45°, and held at 60° and 75°. With order 1, no pooling’s medians (3.4e-03, 4.2e-03, 5.3e-03, 7.2e-03 and 9.3e-03 at 15°, 30°, 45°, 60° and 75°) are below pool C4’s at 15°, 30°, 45° and 60° (6.4e-03, 9.9e-03, 9.3e-03, 6.4e-03) and below pool C8’s at 15°, 30° and 45° (5.0e-03, 7.4e-03, 8.1e-03); it is the largest only at 60° and 75°. With order 3, no pooling’s medians at 15°, 30° and 45° (2.9e-03, 4.2e-03, 6.2e-03) are below pool C4’s (6.1e-03, 1.2e-02, 1.2e-02), and at 15° also below pool C8’s (3.1e-03); it is the largest only at 60° and 75° (8.0e-03, 9.7e-03).
• HB2, pool C4’s median error at 45° exceeds its errors at 15° and 75° and pool C8’s at 45°: held under both orders: 9.3e-03 against 6.4e-03, 5.2e-03 and 8.1e-03 with order 1 (its largest value, 9.9e-03, is at 30°), and 1.2e-02 against 6.1e-03, 2.8e-03 and 5.7e-03 with order 3.
• HB3, true-angle canonicalization has a smaller median error than estimated canonicalization at 15° to 75°: held under order 3 at every angle (6.2e-05 against 6.1e-04 at 15°, 6.1e-05 against 1.9e-04, 9.5e-05 against 5.0e-04, 5.9e-05 against 9.8e-05, 5.8e-05 against 1.9e-03 at 75°). Under order 1 it held at 15°, 30° and 75° (9.7e-04 against 1.5e-03, 9.7e-04 against 1.1e-03, 9.8e-04 against 1.7e-03) and failed at 45° and 60° (1.1e-03 against 8.2e-04, 9.8e-04 against 4.4e-04).
• HB4, cubic interpolation gives smaller errors than bilinear for pool C8 and for true-angle canonicalization at 45°: held (5.7e-03 against 8.1e-03, and 9.5e-05 against 1.1e-03).
• HB5, the median orientation error is below 2° at every angle: failed at 15°, 45° and 75° under both orders (order 1: 3.39°, 3.29°, 3.50°; order 3: 3.32°, 3.02°, 3.51°), and held at 0°, 30°, 60° and 90° (at most 0.60°).

The failed HB1 is the main result of this part. Without any mechanism, the disk-averaged features change by less than 1% (median) at every angle up to 75°, and by 1.1% at 90°; the C4 pool, exact at 90°, changes between quarter turns by amounts of the same order (medians 2.8e-03 to 1.2e-02). In our reading, averaging the responses of many filters over a large textured disk already removes most of the dependence on orientation, because these crops contain edges at many orientations; the averaging is itself an approximate pooling over the rotation group, not a guarantee. The two pools are exact only at the group’s elements, and between them their errors are of the same order as the unpooled average’s. The canonicalization rows show the same thing from the other side. On four of the six crops the second orientation peak is above 0.9 of the highest (camera 0.97, astronaut 0.94, rocket 0.91, coins 0.99), above the 80% at which Lowe creates a second keypoint, and the maximum orientation errors at most angles (82.6° to 178.7°) are of the size a jump to another peak would give, in our reading; yet the canonicalized features never change by more than about 1% (maximum 1e-02), which, in our reading, is because the features they feed are themselves nearly insensitive to orientation. Lowe’s descriptor is different, in our reasoning: its spatial grid of histograms is laid out in the canonical frame, so an orientation error moves every bin; the Feature Descriptors example measures that case.

The orientation errors also show the estimator’s own bias. Rotations by 30° and 60° move the histogram by a whole number of 10° bins, and the median error is at most 0.60°; rotations by 15°, 45° and 75° move it by half a bin more, and the median is 3.02° to 3.51°. In our reading this is the parabola fitted through three bins misplacing peaks that fall between bin centres. Interpolation order matters most where resampling is the only error: rotating back by the true angle costs about $10^{-3}$ with bilinear interpolation and less than $10^{-4}$ with cubic splines.

Colour normalizations under the diagonal-offset model

The third part checks van de Sande et al.’s Table I numerically. The image is coffee, decoded from sRGB to linear values with the standard formula, since the diagonal model describes linear sensor values; its 240,000 pixels are transformed exactly, with no clipping except where stated (the intensity shift can take values above 1, up to 1.04, which the code leaves as they are), and each normalization is computed on the original and the transformed pixels. The normalizations are raw RGB; the chromaticities $r = R/(R+G+B)$ and $g = G/(R+G+B)$; the opponent channels $O_1 = (R - G)/\sqrt{2}$ and $O_2 = (R + G - 2B)/\sqrt{6}$; division of each channel by its image mean (“mean-div”); and the “transformed colour” of van de Sande et al. (their eq. 15), each channel minus its mean and divided by its standard deviation. The measure is the relative change of the normalized pixel array, an exact per-pixel quantity, not a descriptor distance or a retrieval score as in the Global Image Descriptors example. The columns are the five changes of the diagonal-offset model (intensity change by 0.6, intensity shift by 0.04, both, colour change by $(0.9, 0.7, 0.5)$, and colour change with offsets $(0.02, 0.05, 0.08)$), then three changes outside it: mixing of channels by a $3 \times 3$ matrix, a colour change by $(1.6, 1.2, 0.8)$ clipped to $[0, 1]$, and the colour change applied in linear light but normalized on sRGB-encoded values.

# ---- Part D: colour normalizations under the diagonal-offset model ----
def to_linear(c):                           # sRGB decoding (IEC 61966-2-1)
    return np.where(c <= 0.04045, c / 12.92, ((c + 0.055) / 1.055) ** 2.4)

def to_srgb(c):
    return np.where(c <= 0.0031308, 12.92 * c, 1.055 * c ** (1 / 2.4) - 0.055)

rgb = to_linear(data.coffee() / 255.0).reshape(-1, 3)   # linear values, one row per pixel
NORM = {
    "RGB": lambda c: c,
    "rg": lambda c: c[:, :2] / c.sum(1, keepdims=True),
    "O1,O2": lambda c: np.stack([(c[:, 0] - c[:, 1]) / np.sqrt(2),
                                 (c[:, 0] + c[:, 1] - 2 * c[:, 2]) / np.sqrt(6)], 1),
    "mean-div": lambda c: c / c.mean(0),
    "transformed": lambda c: (c - c.mean(0)) / c.std(0),
}
D, O = np.array([0.9, 0.7, 0.5]), np.array([0.02, 0.05, 0.08])
M = np.array([[0.8, 0.15, 0.05], [0.1, 0.8, 0.1], [0.05, 0.15, 0.8]])
TR = {                                       # each maps linear values to linear values
    "int.": lambda c: 0.6 * c,
    "shift": lambda c: c + 0.04,
    "int.+sh.": lambda c: 0.6 * c + 0.04,
    "colour": lambda c: c * D,
    "col.+sh.": lambda c: c * D + O,
    "mixing": lambda c: c @ M.T,
    "clipped": lambda c: np.clip(c * np.array([1.6, 1.2, 0.8]), 0.0, 1.0),
}
print(f"\nPart D data: {len(rgb)} pixels, min R+G+B {rgb.sum(1).min():.1e}, "
      f"pixels clipped in 'clipped' {np.mean((rgb * np.array([1.6, 1.2, 0.8]) > 1).any(1)):.3f}")
print("relative change of each normalization (computed on linear values; last column on sRGB values)")
print(f"{'':12s}" + "".join(f"{t:>10s}" for t in TR) + f"{'colour/sRGB':>13s}")
for n, f in NORM.items():
    row = [rel(f(T(rgb)), f(rgb)) for T in TR.values()]
    row.append(rel(f(to_srgb(rgb * D)), f(to_srgb(rgb))))
    print(f"{n:12s}" + "".join(f"{v:>10.1e}" for v in row[:-1]) + f"{row[-1]:>13.1e}")

Output:

Part D data: 240000 pixels, min R+G+B 3.0e-04, pixels clipped in 'clipped' 0.189
relative change of each normalization (computed on linear values; last column on sRGB values)
                  int.     shift  int.+sh.    colour  col.+sh.    mixing   clipped  colour/sRGB
RGB            4.0e-01   1.2e-01   3.3e-01   2.2e-01   1.9e-01   1.3e-01   4.2e-01      1.3e-01
rg             1.1e-16   2.6e-01   3.1e-01   7.8e-02   3.8e-01   1.6e-01   7.8e-02      7.5e-02
O1,O2          4.0e-01   1.4e-16   4.0e-01   1.6e-01   2.2e-01   2.6e-01   6.5e-01      1.2e-01
mean-div       8.9e-13   2.5e-01   3.5e-01   1.7e-13   4.8e-01   1.9e-01   6.1e-02      3.6e-02
transformed    8.3e-13   6.7e-13   4.7e-13   4.1e-13   2.1e-12   9.5e-02   1.6e-01      1.4e-02

The tolerance for “exact” was set at $10^{-10}$ before the run, because computing means over 240,000 pixels leaves rounding errors of up to about $2 \times 10^{-12}$; “not invariant” means a change of at least $10^{-3}$. Every planned outcome held. The rows for rg, $O_1, O_2$, transformed colour and RGB reproduce Table I as read from the rendered page of the authors’ manuscript: rg is exact only under the intensity change (1.1e-16), $O_1, O_2$ only under the intensity shift (1.4e-16), transformed colour under all five changes (at most 2.1e-12), and RGB under none (at least 1.2e-01). The other cells change by 7.8e-02 or more. Mean division, which the table does not include, is exact under the intensity change (8.9e-13) and the colour change (1.7e-13) and not under any offset (2.5e-01 and more), as its derivation predicts: $aR / \overline{aR} = R / \bar R$ cancels a per-channel factor, while an added constant does not cancel. The same algebra explains the table: $r$ cancels a common factor but not separate ones, $O_1$ and $O_2$ cancel a common offset by subtraction, and subtracting the mean and dividing by the standard deviation cancels any per-channel gain and offset.

Outside the model every normalization fails, as expected. Mixing channels changes all of them by 9.5e-02 or more; in our reasoning, no per-channel operation can undo a change that combines channels. Clipping 18.9% of the pixels breaks the exact invariance of mean division and transformed colour (6.1e-02 and 1.6e-01); van de Sande et al. derive their table on the assumption that no colour is clipped. And a colour change applied to linear light but normalized on sRGB-encoded values changes even the two diagonal-invariant normalizations (3.6e-02 and 1.4e-02), because the sRGB curve, a power law with a linear segment and an offset, does not turn a per-channel factor into a per-channel factor. An exact invariance is exact only for the transformation group and the representation it was derived for.

Common Misconceptions

• “Equivariant means the output transforms the same way as the input.” That is the special case Dieleman et al. call same-equivariance. In general the output transformation can be anything that composes like the group: a lifting layer turns an image rotation into a rotation of each map plus a cyclic shift of the rotation channels (Part A), and a harmonic network into a phase change.
• “A CNN is rotation-equivariant if its filters include rotated copies.” The stack of maps then transforms predictably, but only for the rotations that map the copies onto each other and the grid onto itself, which on a square grid are the quarter turns (Part A exact at 90°, Part B approximate between).
• “Canonicalization makes any descriptor invariant.” Only if the pose estimate is equivariant. A patch with two near-equal orientation peaks gives an unstable estimate; Part B printed orientation errors up to 178.7° on a set of crops in which four had a second peak above 0.9 of the first, and Kaba et al. show that no canonicalization satisfies the standard definition on exactly symmetric inputs.
• “An invariance proved on paper holds on images.” Proofs assume continuous images, a model of the change, and no clipping. Resampling, clipping and gamma encoding each break invariances that the algebra makes exact (Parts B and D).
• “Signal-processing shift invariance and machine-learning shift invariance are the same.” A linear shift-invariant filter commutes with shifts, which this article calls equivariance; a shift-invariant classifier ignores shifts altogether.
• “Invariance is always worth having.” Every invariance discards information about the transformation it removes, and an invariant stage cannot recover it later. The image features and representation learning articles give examples where too much invariance hurts.

Where It Is Used

• Local features. Covariant detectors and invariant descriptors in SIFT, ORB and their relatives, for feature matching across viewpoints, scales and lighting.
• Convolutional networks. Translation equivariance from weight sharing and invariance from global pooling, the inductive bias every CNN carries; group-equivariant and harmonic networks for data without a preferred orientation, in the fields Dieleman et al. name: biology, astronomy, medicine and aerial photography.
• Learned representations. Invariances chosen through augmentation in supervised and self-supervised learning, and measured after training with the methods of Lenc and Vedaldi.
• Texture and shape. Rotation-invariant local binary patterns, moment invariants for isolated shapes, and multi-scale scale-space representations for scale.
• Colour. Chromaticity, opponent and normalized colour spaces in descriptors and segmentation; the color spaces article describes the spaces themselves.
• Geometric structures beyond the grid. Permutation equivariance in graph neural networks and self-attention, and rotation handling in 3D deep learning.