Evaluating Representations

Evaluating a representation means measuring how useful the features of a pretrained, usually frozen, encoder are for tasks it was not trained on. A cheap readout, called a protocol, is fitted on labelled downstream images and scored on held-out ones. The common protocols are a linear classifier on the frozen features, a nearest-neighbour vote, low-shot versions of both, and fine-tuning of the whole network; benchmark suites run them over many datasets. Representation learning summarizes these protocols and shows that they can rank encoders differently. This article covers them in depth: how the papers that set them define them, which solver settings and hyperparameters move their results, which changes of the feature coordinates each one ignores, and how a closed-form probe can select its own regularization exactly, without a validation split.

Definition

An evaluation protocol takes a fixed encoder $f$, a labelled downstream training set and a test set, and returns a score, usually top-1 accuracy of the classifier it builds on $f$. A frozen-feature protocol computes the features once and fits only a readout: a linear classifier (a linear probe), a least-squares map to class targets, or no trained readout at all, as in k-nearest-neighbour (k-NN) classification. Fine-tuning also changes the encoder, so it judges the pretrained weights as a starting point for training. A low-shot protocol restricts the downstream training set to a few labelled images per class.

The protocol is part of the measurement. In our reading, two encoders evaluated under different protocols, solvers, layers or hyperparameter searches are not directly comparable; Goyal, Mahajan, Gupta and Misra (2019) found inconsistencies across earlier self-supervised evaluations, including different image crops, weight rescaling, pre-processing and longer fine-tuning schedules, which they report affected the final performance.

Intuition

A frozen representation is a set of coordinates for images. A protocol asks a question about those coordinates: can the classes be separated by a hyperplane (linear probe), are images of one class close to each other (k-NN), can a few labels pin down a classifier (low-shot), or can training start from here and go further (fine-tuning)? Each question has blind spots and sensitivities. A hyperplane can be found after any invertible linear change of coordinates, but a hyperplane with a penalty on its weights prefers directions in which the features are large. A nearest-neighbour vote depends on what “close” means, so stretching one coordinate can change every neighbour. Two encoders whose features differ only by such a change of coordinates carry the same information, yet most protocols score them differently.

So a protocol’s score mixes three things: what the encoder has made available, how the protocol’s readout matches the form in which it is available, and the protocol’s own settings. The rest of this article separates them.

Formal Definition

Let $f_\theta : \mathcal{X} \to \mathbb{R}^d$ be the encoder, $\{(x_i, y_i)\}_{i=1}^{n}$ the labelled downstream training images with $y_i \in \{1, \dots, K\}$, and $h_i = f_\theta(x_i)$ their features, stacked as the rows of $H \in \mathbb{R}^{n \times d}$.

Logistic linear probe. A multinomial logistic regression on the frozen features,

$$\min_{W,\, b}\; \sum_{i=1}^{n} \ell_{\text{CE}}\big(W h_i + b,\; y_i\big) + \lambda \lVert W \rVert_2^2 ,$$

with $\ell_{\text{CE}}$ the cross-entropy of a softmax. The problem is convex. Kornblith, Shlens and Le (2019) solved it with L-BFGS on penultimate-layer features without data augmentation, with the $\ell_2$ weight applied to the sum of the per-example losses and selected on a validation set from 45 logarithmically spaced values between $10^{-6}$ and $10^{5}$, warm-starting each value from the previous one (supplement, A.4). Kolesnikov, Zhai and Beyer (2019) also used L-BFGS on precomputed features for their large study, and switched to stochastic gradient descent with momentum and data augmentation when comparing with published numbers (§3.3). Caron et al. (2021) trained the DINO linear classifier with random resized crops and flips and tested on a central crop. Goyal et al. (2019) trained linear support vector machines with LIBLINEAR on VOC07 and COCO2014, and followed an SGD schedule from earlier work for Places205 (§6.1).

Least-squares probe. With one-vs-rest targets $T \in \{-1, 1\}^{n \times K}$ ($+1$ in the column of the true class), the regularized least-squares map has a closed form. With an unpenalized bias, write $Z = [\mathbf{1}, H]$ and $P = \operatorname{diag}(0, \lambda I_d)$; then

$$B = (Z^\top Z + P)^{-1} Z^\top T, \qquad \hat{y}(h) = \arg\max_k \big([1, h^\top] B\big)_k .$$

Without a bias, $Z = H$ and $P = \lambda I_d$. This is the linear few-shot evaluation of the ViT paper, which, as that article describes it, maps the frozen features of a few training images per class to $\{-1, 1\}^K$ targets by regularized least squares, solved in closed form, and was used where fine-tuning every model was too costly. The article does not say whether the map has a bias, so both forms appear below. With $\lambda = 0$ it is plain least squares.

Weighted k-NN. Wu, Xiong, Yu and Lin (2018, §3.4) compare a test feature $h$ with every stored training feature by the cosine $s_i = \cos(h_i, h)$, keep the $k$ most similar, $\mathcal{N}_k$, and give class $c$ the weight

$$w_c = \sum_{i \in \mathcal{N}_k} \exp(s_i / \tau)\, \mathbf{1}(y_i = c),$$

predicting the class with the largest weight. They used $\tau = 0.07$, as in training, and $k = 200$. Caron et al. (2021) adopted this classifier for DINO and found, sweeping $k$, that 20 neighbours worked best for most of their runs. A plain k-NN vote with Euclidean distance is the unweighted special case with another metric.

Fine-tuning. The same downstream loss is minimized over the encoder’s parameters $\theta$ as well, starting from the pretrained values with a new head; fine-tuning covers its variants. Kornblith et al. (2019) fine-tuned for 20,000 steps at batch size 256 with Nesterov momentum 0.9, choosing learning rate and weight decay by grid search on a validation set (supplement, A.5).

Invariance of a protocol. Let $g : \mathbb{R}^d \to \mathbb{R}^d$ be invertible. The representation $g \circ f_\theta$ carries exactly the same information about the image as $f_\theta$. A protocol $\mathcal{E}$ is invariant to $g$ if the classifier it fits on $\{(g(h_i), y_i)\}$ predicts at $g(h)$ the same class that the classifier it fits on $\{(h_i, y_i)\}$ predicts at $h$, for every training set and every test point. The transformations considered below act on feature vectors, not on images; invariance and equivariance treats the image-side notion.

Properties

What each protocol ignores

Which changes of coordinates a protocol ignores follows from its definition. The derivations below are ours; the example checks each of them numerically.

Least squares with a bias is invariant to every invertible affine map $g(h) = M h + c$. The design matrix becomes $Z' = Z G$ with

$$G = \begin{pmatrix} 1 & c^\top \\ 0 & M^\top \end{pmatrix},$$

which is invertible, so the fitted coefficients become $G^{-1} B$ and the test scores $[1, h^\top] G\, G^{-1} B$ are unchanged. The same holds for any unregularized model that is linear in $[1, h]$, such as logistic regression without a penalty, when its optimum exists.

A penalty breaks this. For the regularized problem the coefficients become $G^{-1}(Z^\top Z + G^{-\top} P\, G^{-1})^{-1} Z^\top T$, so the predictions are unchanged for every training set only when $G^{-\top} P\, G^{-1} = P$. With $P = \operatorname{diag}(0, \lambda I)$ this reduces to $M^{-1} M^{-\top} = I$: $M$ must be orthogonal, and the offset $c$ is free because the bias absorbs it. Without a bias, $c$ must be zero. An isotropic scale $M = aI$ is equivalent to the same probe with $\lambda / a^2$. The $\ell_2$-penalized logistic probe has the same invariances at its optimum, since its penalty is the same; a solver reaches the optimum only to its tolerance.

Euclidean k-NN depends only on the ranking of distances, which rotations and translations preserve and an isotropic scale multiplies by a constant. Cosine k-NN depends on angles, which rotations and positive scales preserve and a translation changes. Per-feature scaling and general linear maps change both.

Preprocessing moves the invariances. Standardizing each feature with training means and standard deviations undoes any translation and any positive per-feature scaling, so a protocol run after it inherits that invariance, but loses invariance to rotation, since rotated features have other per-feature statistics. Whitening with the training covariance $\Sigma$ maps $g(h)$ to $R\,\Sigma^{-1/2}(h - \mu)$ with $R = (M \Sigma M^\top)^{-1/2} M \Sigma^{1/2}$, and $R R^\top = I$; a rotation-invariant protocol run after whitening is therefore invariant to every invertible affine map. Euclidean k-NN on whitened features is k-NN with the Mahalanobis distance of the training covariance (similarity and distance measures). Kornblith et al. (2019, supplement G.1) $\ell_2$-normalized features before their SVMs, following earlier work; by the same reasoning, that makes the SVM ignore any positive rescaling of a single feature vector.

Protocol	Rotation	Isotropic scale	Translation	Per-feature scale	Any invertible affine map
Least squares, with bias, $\lambda = 0$	yes	yes	yes	yes	yes
Ridge or $\ell_2$ logistic, no bias	yes	no	no	no	no
Ridge or $\ell_2$ logistic, unpenalized bias	yes	no	yes	no	no
Euclidean k-NN	yes	yes	yes	no	no
Cosine (weighted) k-NN	yes	yes	no	no	no
Standardize, then ridge	no	yes	yes	yes	no
Whiten, then Euclidean k-NN or ridge	yes	yes	yes	yes	yes

Invariance is not accuracy. A protocol that ignores every invertible map cannot prefer a representation for presenting the classes in a convenient form, and in practice a representation is judged partly on that form; the example shows one of its two fully invariant protocols, whitened 5-NN, scoring lowest.

Settings, solvers and hyperparameters

Every protocol has settings, and their influence is reported in the papers that set the protocols. Kolesnikov et al. (2019, §4.7) trained the linear evaluation with SGD at batch size 2,048, decaying the learning rate twice, and found that when the first decay came had a large influence on the final accuracy: accuracy kept improving after very long training (about 500 epochs), so they concluded that the optimization hyperparameters need to be reported. Caron et al. (2021) report that linear and fine-tuning evaluations are sensitive to hyperparameters, with large variance between runs when the learning rate is varied, which is why they added the weighted k-NN, which needs no further tuning, no augmentation and a single pass over the downstream data. k-NN still has settings: the metric, the normalization, $k$ (200 for Wu et al., 20 for DINO) and $\tau$. Kornblith et al. (2019) selected logistic-regression regularization on a validation set, and repeated the study with SVMs on $\ell_2$-normalized features and with logistic regression trained with augmentation; their findings did not change (supplement G).

A convex probe removes the dependence on initialization, not on the solver. L-BFGS on precomputed features, as Kornblith et al. and Kolesnikov et al. used it, stops at a tolerance or an evaluation limit; in the example below, it stopped before its tolerance on four of five fits to badly scaled features.

Exact leave-one-out for closed-form probes

The least-squares probe can select $\lambda$ without a validation split. With the hat matrix $\mathcal{H} = Z (Z^\top Z + P)^{-1} Z^\top$, the leave-one-out residual of training image $i$ is

$$e_{(i)} = \frac{t_i - (\mathcal{H} T)_i}{1 - \mathcal{H}_{ii}},$$

so the leave-one-out scores $t_i - e_{(i)}$ for all $n$ images, and the predicted residual sum of squares $\text{PRESS}(\lambda) = \frac{1}{nK}\sum_i \lVert e_{(i)} \rVert^2$, cost one solve instead of $n$ refits. The identity follows from the Sherman–Morrison formula for removing one row from $Z^\top Z + P$, because the penalty does not depend on the data. It also implies $\text{PRESS}(a h, a^2 \lambda) = \text{PRESS}(h, \lambda)$ for an isotropic scale $a$: selecting $\lambda$ by leave-one-out on a logarithmic grid absorbs a global rescaling of the features, as long as the grid is wide enough, but not a per-feature one. In our reading, this suits the low-shot regime, where a held-out split would take labels away from a few images per class. Cross-validation covers leave-one-out estimation in general.

Layer, readout and label budget

Which layer. The best layer to read out as deep features depends on the architecture. Kolesnikov et al. (2019, §4.5) found that in their self-supervised VGG19-BN the representation quality deteriorated towards the end of the network, while in models with skip connections it increased up to the final pre-logits layer. Goyal et al. (2019) extract features after every convolutional layer of AlexNet and every residual stage of ResNet-50, and report results for each layer.

How expressive a readout. Kolesnikov et al. (2019, §4.3) replaced the linear model with a multilayer perceptron with one hidden layer of 1,000 units and found only a marginal improvement, with the relative performance of settings mostly unchanged, so they concluded that the linear model is adequate for their purposes. Other readouts sit between a linear probe and full fine-tuning: attentive probes that pool token features by learned cross-attention (video foundation models), fine-tuning only the last blocks (masked autoencoder, whose authors argue from it that linear separability is not the only measure of representation quality), probing before fine-tuning (fine-tuning), and the selective and additive methods of parameter-efficient fine-tuning.

How many labels. Goyal et al. (2019, §6.2) trained linear SVMs with up to 96 (VOC07) and 128 (Places205) positive examples per class, reporting the mean and standard deviation over five independent samples of the training data on a fixed test set. Low-shot evaluation measures how well a few labels specify the classifier, which full-data probes do not.

Benchmarks and disagreement between protocols

A single downstream dataset rewards representations close to it. Goyal et al. (2019) proposed a suite of nine tasks, from image classification on Places205, VOC07 and COCO2014 and low-shot classification to visual navigation, object detection and surface-normal estimation, built on the principle that a good representation transfers to many tasks with limited supervision and limited fine-tuning. They argued for a consistent evaluation setup, with hyperparameters set fairly for all methods, to separate better representations from better transfer optimization, and left ImageNet linear classification out of the suite because, as one of two reasons, evaluating a supervised ImageNet model on ImageNet does not test its generalization (§6, §7). Representation learning describes the Visual Task Adaptation Benchmark and the cases in which linear and fine-tuned results disagreed.

Protocols also disagree on single datasets. On CIFAR-10, Wu et al. (2018, Table 1) report 60.3% with a linear SVM and 63.0% with k-NN on features from a parametric softmax, against 75.4% and 80.8% for their non-parametric instance discrimination; they argue that the nearest-neighbour classifier relies directly on the feature metric and may better reflect the quality of the representation. Kornblith et al. (2019) found that fine-tuning beat logistic regression on fixed features in 179 of 192 dataset–model combinations, and still beat logistic regression trained with augmentation in 188 of 192 (supplement G.2). Kolesnikov et al. (2019) found that neither the ranking of architectures was consistent across self-supervised methods, nor the ranking of methods across architectures (§4.1), and that pretext-task accuracy predicted downstream accuracy only once the architecture was fixed (§4.4).

Examples

The script below computes a fixed representation of synthetic textures and asks which protocols notice exact changes of its coordinates. The images are $32 \times 32$ grey textures of four classes of scattered elements (disks, rings, bars and crosses at random orientation), 5 to 9 elements per image with random size, position and contrast on a random background level, plus Gaussian pixel noise of standard deviation 0.05. The representation is 32 random $5 \times 5$ filters with zero mean and unit norm, a ReLU and global average pooling: nothing is trained. Each data seed draws 25 training and 250 test images per class. Five fixed maps are applied to the training and test features alike: a random rotation, a scale by 10, a translation by a random vector, a per-feature (“diagonal”) scale by factors $10^{u}$ with $u$ uniform on $[-2, 2]$, and a dense invertible matrix with singular values $10^{v}$, $v$ uniform on $[-1, 1]$.

Eight protocols are compared: least squares to $\{-1, 1\}^4$ targets with a bias; the ViT-style ridge probe with $\lambda = 10^{-4}$, without and with an unpenalized bias, since the bias is an open modelling choice; $\ell_2$ logistic regression (scikit-learn, L-BFGS, $C = 10^6$, tolerance $10^{-10}$); Euclidean 5-NN; the weighted cosine 20-NN of Wu et al. with DINO’s $k$ and $\tau = 0.07$; ridge with $\lambda = 1$ after standardization; and Euclidean 5-NN after whitening. Part B computes the ridge probe’s leave-one-out scores through the hat matrix, checks them against 100 explicit refits, and then re-selects $\lambda$ by minimum PRESS on each transformed feature set. The noise level, the fixed ridge $\lambda = 10^{-4}$ (the leave-one-out optimum), $C = 10^6$ (5-fold cross-validation over $10^0$ to $10^8$) and the standardized-ridge $\lambda = 1$ (leave-one-out) were chosen on the original features of seed 0 before the example was first run; after a trial run, scikit-learn’s convergence warnings were recorded in the code instead of being shown, and the line that counts logistic fits stopping before their tolerance was added; nothing else changed. The script loops over data seeds 0 to 4, runs in about 10 seconds on one CPU thread, and printed identical output on two runs with NumPy 2.3.5 and scikit-learn 1.7.2.

import os
for v in ("OPENBLAS_NUM_THREADS", "OMP_NUM_THREADS", "MKL_NUM_THREADS", "VECLIB_MAXIMUM_THREADS"):
    os.environ[v] = "1"                      # one BLAS thread: identical sums on every run
import numpy as np
import warnings
import sklearn
from sklearn.exceptions import ConvergenceWarning
from numpy.lib.stride_tricks import sliding_window_view
from sklearn.linear_model import LinearRegression, LogisticRegression, Ridge
from sklearn.neighbors import KNeighborsClassifier

S, K, d = 32, 4, 32                          # image size, classes, feature dimension
yy, xx = np.mgrid[0:S, 0:S].astype(float)

def element(kind, cy, cx, r, th):
    """Mask of one texture element: 0 disk, 1 ring, 2 bar, 3 cross (r = half-size)."""
    dy, dx = yy - cy, xx - cx
    u, v = np.cos(th) * dx + np.sin(th) * dy, -np.sin(th) * dx + np.cos(th) * dy
    if kind == 0: return dy**2 + dx**2 <= r**2
    if kind == 1: return (np.hypot(dy, dx) <= r) & (np.hypot(dy, dx) >= r - 1.2)
    bar = (np.abs(u) <= r) & (np.abs(v) <= 0.7)
    return bar if kind == 2 else bar | ((np.abs(v) <= r) & (np.abs(u) <= 0.7))

def texture(kind, rng, log):
    """32x32 texture: 5-9 elements of one kind on a flat background, plus noise."""
    img = np.full((S, S), rng.uniform(0.3, 0.5))
    n = rng.integers(5, 10)
    for _ in range(n):
        r = rng.uniform(2.0, 4.0)
        cy, cx = rng.uniform(r, S - 1 - r, 2)
        m = element(kind, cy, cx, r, rng.uniform(0, np.pi))
        img[m] += rng.uniform(0.2, 0.6)
        log.append((n, m.sum()))
    return img + rng.normal(0, 0.05, (S, S))

def dataset(n_per_class, rng, log):
    y = np.repeat(np.arange(K), n_per_class)
    return np.stack([texture(k, rng, log) for k in y]), y

W = np.random.default_rng(1000).normal(size=(d, 5, 5))     # fixed random 5x5 filters
W -= W.mean((1, 2), keepdims=True)
W /= np.linalg.norm(W.reshape(d, -1), axis=1)[:, None, None]
def features(X):
    """Frozen representation: random filters, ReLU, global average pooling -> 32-d."""
    win = sliding_window_view(X, (5, 5), axis=(1, 2))
    return np.maximum(np.einsum("nijab,fab->nijf", win, W), 0).mean((1, 2))

def orth(seed):
    q, r = np.linalg.qr(np.random.default_rng(seed).normal(size=(d, d)))
    return q * np.sign(np.diag(r))
Q = orth(104)
b = np.random.default_rng(101).normal(0, 0.05, d)
D = 10 ** np.random.default_rng(103).uniform(-2, 2, d)
A = orth(105) @ np.diag(10 ** np.random.default_rng(102).uniform(-1, 1, d)) @ orth(106).T
TRANSFORMS = {"rotation": lambda F: F @ Q.T, "scale x10": lambda F: 10 * F,
              "translation": lambda F: F + b, "diagonal": lambda F: F * D,
              "dense": lambda F: F @ A.T}

# ---- protocols: each returns class scores for the test features ----
def ridge(Ftr, ytr, Fte, lam, bias=True):
    """Least squares to {-1, 1}^K targets; lam = 0 is plain least squares."""
    Y = 2 * np.eye(K)[ytr] - 1
    Z, Zt = (np.c_[np.ones(len(Ftr)), Ftr], np.c_[np.ones(len(Fte)), Fte]) if bias else (Ftr, Fte)
    P = lam * np.eye(Z.shape[1])
    if bias: P[0, 0] = 0.0                                      # intercept not penalized
    return Zt @ np.linalg.solve(Z.T @ Z + P, Z.T @ Y)

def knn(Ftr, ytr, Fte, k=5):
    D2 = (Fte**2).sum(1)[:, None] - 2 * Fte @ Ftr.T + (Ftr**2).sum(1)[None]
    nn = np.argsort(D2, 1, kind="stable")[:, :k]
    return np.stack([(ytr[nn] == c).sum(1) for c in range(K)], 1).astype(float)

def cosine_knn(Ftr, ytr, Fte, k=20, T=0.07):
    """Weighted k-NN of Wu et al. (2018) with DINO's k = 20: weights exp(cos / T)."""
    s = (Fte / np.linalg.norm(Fte, axis=1, keepdims=True)) @ (Ftr / np.linalg.norm(Ftr, axis=1, keepdims=True)).T
    nn = np.argsort(-s, 1, kind="stable")[:, :k]
    w = np.exp(np.take_along_axis(s, nn, 1) / T)
    return np.stack([(w * (ytr[nn] == c)).sum(1) for c in range(K)], 1)

def standardize(Ftr, Fte):
    mu, sd = Ftr.mean(0), Ftr.std(0)
    return (Ftr - mu) / sd, (Fte - mu) / sd

def whiten(Ftr, Fte):
    w, E = np.linalg.eigh(np.cov(Ftr, rowvar=False))
    M = E @ np.diag(w ** -0.5) @ E.T
    return (Ftr - Ftr.mean(0)) @ M, (Fte - Ftr.mean(0)) @ M

def then(prep, protocol, Ftr, ytr, Fte, *args):
    """Apply a preprocessing fitted on the training features, then a protocol."""
    a, t = prep(Ftr, Fte)
    return protocol(a, ytr, t, *args)

LAM, LAM_STD, C_LOGREG = 1e-4, 1.0, 1e6      # fixed once on the original features of seed 0
stopped, CURRENT = {}, "original"           # L2-logistic fits that stop before tol, per feature set
def logistic(Ftr, ytr, Fte):
    with warnings.catch_warnings(record=True) as caught:
        warnings.simplefilter("always", ConvergenceWarning)
        m = LogisticRegression(C=C_LOGREG, tol=1e-10, max_iter=100000).fit(Ftr, ytr)
    if any(issubclass(w.category, ConvergenceWarning) for w in caught):
        stopped[CURRENT] = stopped.get(CURRENT, 0) + 1
    return m.decision_function(Fte)
PROTOCOLS = {
    "LS (lambda=0), bias":   lambda a, y, t: ridge(a, y, t, 0.0),
    "ridge, no bias":        lambda a, y, t: ridge(a, y, t, LAM, bias=False),
    "ridge, bias":           lambda a, y, t: ridge(a, y, t, LAM),
    "L2 logistic":           lambda a, y, t: logistic(a, y, t),
    "5-NN Euclidean":        knn,
    "weighted 20-NN cosine": cosine_knn,
    "standardize + ridge":   lambda a, y, t: then(standardize, ridge, a, y, t, LAM_STD),
    "whiten + 5-NN":         lambda a, y, t: then(whiten, knn, a, y, t),
}

def loo(Ftr, ytr, lam):
    """Exact leave-one-out scores and PRESS of the ridge probe with bias, via the hat matrix."""
    Y = 2 * np.eye(K)[ytr] - 1
    Z = np.c_[np.ones(len(Ftr)), Ftr]
    P = lam * np.eye(d + 1); P[0, 0] = 0.0
    H = Z @ np.linalg.solve(Z.T @ Z + P, Z.T)
    E = (Y - H @ Y) / (1 - np.diag(H))[:, None]
    return Y - E, (E**2).mean()

GRID = 10.0 ** np.arange(-8, 5)
names, tnames = list(PROTOCOLS), list(TRANSFORMS)
changed = np.zeros((5, len(names), len(tnames)), int)
dscore = np.zeros((len(names), len(tnames)))
acc = np.zeros((5, len(names), len(tnames) + 1))
partB = {t: [] for t in ["original"] + tnames}
loo_err, press_err, log, counts_ok = 0.0, 0.0, [], True
for seed in range(5):
    rng = np.random.default_rng(seed)
    Xtr, ytr = dataset(25, rng, log); Xte, yte = dataset(250, rng, log)
    counts_ok = counts_ok and set(np.bincount(ytr)) == {25} and set(np.bincount(yte)) == {250}
    Ftr, Fte = features(Xtr), features(Xte)
    CURRENT = "original"
    base = {p: f(Ftr, ytr, Fte) for p, f in PROTOCOLS.items()}
    for i, p in enumerate(names):
        acc[seed, i, 0] = np.mean(base[p].argmax(1) == yte)
    for j, t in enumerate(tnames):
        Gtr, Gte = TRANSFORMS[t](Ftr), TRANSFORMS[t](Fte)
        CURRENT = t
        for i, p in enumerate(names):
            s = PROTOCOLS[p](Gtr, ytr, Gte)
            changed[seed, i, j] = np.sum(s.argmax(1) != base[p].argmax(1))
            dscore[i, j] = max(dscore[i, j], np.abs(s - base[p]).max())
            acc[seed, i, j + 1] = np.mean(s.argmax(1) == yte)
    # Part B: exact LOO, then lambda re-selected by minimum PRESS on each feature set
    L, _ = loo(Ftr, ytr, LAM)
    refit = np.stack([ridge(np.delete(Ftr, i, 0), np.delete(ytr, i), Ftr[i:i + 1], LAM)[0] for i in range(len(ytr))])
    loo_err = max(loo_err, np.abs(L - refit).max())
    p1 = np.array([loo(Ftr, ytr, l)[1] for l in GRID])
    p10 = np.array([loo(10 * Ftr, ytr, 100 * l)[1] for l in GRID])
    press_err = max(press_err, np.max(np.abs(p10 - p1) / p1))
    ref = None
    for t in ["original"] + tnames:
        Gtr, Gte = (Ftr, Fte) if t == "original" else (TRANSFORMS[t](Ftr), TRANSFORMS[t](Fte))
        lam = GRID[np.argmin([loo(Gtr, ytr, l)[1] for l in GRID])]
        pred = ridge(Gtr, ytr, Gte, lam).argmax(1)
        ref = pred if t == "original" else ref
        partB[t].append((lam, np.mean(pred == yte), np.sum(pred != ref)))

ns, areas = np.array(log).T
print(f"NumPy {np.__version__}, scikit-learn {sklearn.__version__}")
print(f"data: 25 training and 250 test images per class in every seed: {counts_ok}; "
      f"elements per image {ns.min()}-{ns.max()}; smallest element {areas.min()} px")
w = 23
print("\nA1. changed test predictions (of 1,000), mean [min-max] over seeds 0-4")
print(" " * w + "".join(f"{t:>16s}" for t in tnames))
for i, p in enumerate(names):
    c = changed[:, i, :]
    print(f"{p:{w}s}" + "".join(f"{m:7.1f} [{lo:3d}-{hi:3d}]" for m, lo, hi in zip(c.mean(0), c.min(0), c.max(0))))
print("\nA2. largest |change in class scores| over test images and seeds")
print(" " * w + "".join(f"{t:>12s}" for t in tnames))
for i, p in enumerate(names):
    print(f"{p:{w}s}" + "".join(f"{v:12.1e}" for v in dscore[i]))
print("\nA3. test accuracy: original mean [min-max]; transformed, mean over seeds")
print(" " * w + f"{'original':>19s}" + "".join(f"{t:>12s}" for t in tnames))
for i, p in enumerate(names):
    a = acc[:, i, :]
    print(f"{p:{w}s}{a[:, 0].mean():8.3f} [{a[:, 0].min():.3f}-{a[:, 0].max():.3f}]"
          + "".join(f"{v:12.3f}" for v in a[:, 1:].mean(0)))
print("L2-logistic fits stopped before tol, ConvergenceWarning (seeds of 5): "
      + ", ".join(f"{t} {stopped.get(t, 0)}" for t in ["original"] + tnames))
print(f"\nB. ridge with bias: max |hat-matrix LOO - 100 refits| {loo_err:.1e}; "
      f"max rel. |PRESS(10h, 100 lam) - PRESS(h, lam)| {press_err:.1e}")
print("   lambda re-selected by minimum PRESS on each feature set")
for t, r in partB.items():
    lam, a, ch = np.array(r).T
    print(f"   {t:12s} lambda {' '.join(f'{l:.0e}' for l in lam)}  acc {a.mean():.3f} [{a.min():.3f}-{a.max():.3f}]"
          f"  changed vs original {ch.mean():5.1f} [{int(ch.min())}-{int(ch.max())}]")

# scikit-learn cross-checks on seed 0
rng = np.random.default_rng(0)
Xtr, ytr = dataset(25, rng, []); Xte, yte = dataset(250, rng, [])
Ftr, Fte = features(Xtr), features(Xte)
Y = 2 * np.eye(K)[ytr] - 1
VI = np.linalg.inv(np.cov(Ftr, rowvar=False))
print("\nseed-0 checks against scikit-learn:")
print(f"  LS vs LinearRegression {np.abs(ridge(Ftr, ytr, Fte, 0.0) - LinearRegression().fit(Ftr, Y).predict(Fte)).max():.1e}; "
      f"ridge vs Ridge {np.abs(ridge(Ftr, ytr, Fte, LAM) - Ridge(alpha=LAM).fit(Ftr, Y).predict(Fte)).max():.1e}; "
      f"no bias {np.abs(ridge(Ftr, ytr, Fte, LAM, False) - Ridge(alpha=LAM, fit_intercept=False).fit(Ftr, Y).predict(Fte)).max():.1e}")
print("  5-NN identical:", np.array_equal(knn(Ftr, ytr, Fte).argmax(1),
                                         KNeighborsClassifier(5, algorithm="brute").fit(Ftr, ytr).predict(Fte)),
      "; whitened 5-NN = Mahalanobis 5-NN:", np.array_equal(PROTOCOLS["whiten + 5-NN"](Ftr, ytr, Fte).argmax(1),
      KNeighborsClassifier(5, algorithm="brute", metric="mahalanobis", metric_params={"VI": VI}).fit(Ftr, ytr).predict(Fte)))

NumPy 2.3.5, scikit-learn 1.7.2
data: 25 training and 250 test images per class in every seed: True; elements per image 5-9; smallest element 4 px

A1. changed test predictions (of 1,000), mean [min-max] over seeds 0-4
                               rotation       scale x10     translation        diagonal           dense
LS (lambda=0), bias        0.0 [  0-  0]    0.0 [  0-  0]    0.0 [  0-  0]    0.0 [  0-  0]    0.0 [  0-  0]
ridge, no bias             0.0 [  0-  0]  154.6 [129-183]  171.0 [142-190]  247.0 [205-299]  138.2 [119-163]
ridge, bias                0.0 [  0-  0]  124.6 [ 98-154]    0.0 [  0-  0]  188.8 [154-249]   93.4 [ 74-118]
L2 logistic                0.0 [  0-  0]   34.8 [ 21- 50]    0.0 [  0-  0]  189.0 [155-238]  118.6 [ 96-127]
5-NN Euclidean             0.0 [  0-  0]    0.0 [  0-  0]    0.0 [  0-  0]  323.0 [293-347]  159.8 [146-177]
weighted 20-NN cosine      0.0 [  0-  0]    0.0 [  0-  0]  462.6 [441-479]  421.0 [374-474]  201.0 [181-229]
standardize + ridge       86.4 [ 45-124]    0.0 [  0-  0]    0.0 [  0-  0]    0.0 [  0-  0]  119.2 [ 88-145]
whiten + 5-NN              0.0 [  0-  0]    0.0 [  0-  0]    0.0 [  0-  0]    0.0 [  0-  0]    0.0 [  0-  0]

A2. largest |change in class scores| over test images and seeds
                           rotation   scale x10 translation    diagonal       dense
LS (lambda=0), bias         5.0e-11     4.6e-11     1.0e-10     3.5e-11     3.8e-09
ridge, no bias              3.9e-12     2.0e+00     2.0e+00     1.9e+00     1.3e+00
ridge, bias                 3.3e-12     1.9e+00     9.4e-12     2.0e+00     1.0e+00
L2 logistic                 1.2e-03     6.8e+01     1.8e-03     7.4e+01     6.4e+01
5-NN Euclidean              0.0e+00     0.0e+00     0.0e+00     5.0e+00     4.0e+00
weighted 20-NN cosine       1.3e-07     1.5e-07     2.7e+07     3.0e+07     1.7e+07
standardize + ridge         1.1e+00     1.7e-13     1.8e-13     1.8e-13     1.0e+00
whiten + 5-NN               0.0e+00     0.0e+00     0.0e+00     0.0e+00     0.0e+00

A3. test accuracy: original mean [min-max]; transformed, mean over seeds
                                  original    rotation   scale x10 translation    diagonal       dense
LS (lambda=0), bias       0.800 [0.738-0.850]       0.800       0.800       0.800       0.800       0.800
ridge, no bias            0.771 [0.730-0.797]       0.771       0.754       0.832       0.673       0.746
ridge, bias               0.831 [0.805-0.860]       0.831       0.802       0.831       0.737       0.814
L2 logistic               0.871 [0.839-0.912]       0.871       0.858       0.871       0.775       0.824
5-NN Euclidean            0.719 [0.702-0.742]       0.719       0.719       0.719       0.570       0.704
weighted 20-NN cosine     0.769 [0.746-0.792]       0.769       0.769       0.658       0.518       0.735
standardize + ridge       0.824 [0.803-0.852]       0.815       0.824       0.824       0.824       0.800
whiten + 5-NN             0.590 [0.568-0.646]       0.590       0.590       0.590       0.590       0.590
L2-logistic fits stopped before tol, ConvergenceWarning (seeds of 5): original 0, rotation 0, scale x10 0, translation 0, diagonal 4, dense 0

B. ridge with bias: max |hat-matrix LOO - 100 refits| 1.1e-12; max rel. |PRESS(10h, 100 lam) - PRESS(h, lam)| 9.1e-13
   lambda re-selected by minimum PRESS on each feature set
   original     lambda 1e-04 1e-04 1e-04 1e-04 1e-04  acc 0.831 [0.805-0.860]  changed vs original   0.0 [0-0]
   rotation     lambda 1e-04 1e-04 1e-04 1e-04 1e-04  acc 0.831 [0.805-0.860]  changed vs original   0.0 [0-0]
   scale x10    lambda 1e-02 1e-02 1e-02 1e-02 1e-02  acc 0.831 [0.805-0.860]  changed vs original   0.0 [0-0]
   translation  lambda 1e-04 1e-04 1e-04 1e-04 1e-04  acc 0.831 [0.805-0.860]  changed vs original   0.0 [0-0]
   diagonal     lambda 1e-06 1e-07 1e-07 1e-06 1e-07  acc 0.799 [0.750-0.838]  changed vs original 119.6 [75-164]
   dense        lambda 1e-04 1e-04 1e-04 1e-06 1e-04  acc 0.816 [0.792-0.843]  changed vs original  95.2 [74-127]

seed-0 checks against scikit-learn:
  LS vs LinearRegression 1.4e-11; ridge vs Ridge 2.1e-12; no bias 1.2e-12
  5-NN identical: True ; whitened 5-NN = Mahalanobis 5-NN: True

The data line confirms the class counts, 5 to 9 elements per image and a smallest element of 4 pixels. Least squares on the original features reached 0.738 to 0.850 over the seeds, far above the chance level of 0.25 but not perfect, so the fixed features carry the classes without making them trivial. Table A1 confirms the invariance table cell by cell. Every cell the derivation calls invariant changed no prediction in any seed: least squares in all five columns; both ridge probes under rotation; ridge with a bias and $\ell_2$ logistic under translation; Euclidean 5-NN under rotation, scale and translation; cosine 20-NN under rotation and scale; the standardized ridge under scale, translation and diagonal scaling; and whitened 5-NN in all five columns. Table A2 shows what kind of invariance each is. The closed-form probes agree to rounding error, at most $3.8 \times 10^{-9}$ in a class score (least squares under the dense map) and $10^{-10}$ to $10^{-13}$ elsewhere; the two Euclidean k-NN rows give identical votes; the cosine votes, sums of terms $\exp(s/0.07)$ that can reach $e^{1/0.07} \approx 1.6 \times 10^{6}$, differ by about $10^{-7}$. The $\ell_2$ logistic probe is invariant to rotation and translation only to the solver’s tolerance, with score changes of $1.2 \times 10^{-3}$ and $1.8 \times 10^{-3}$, which in this run moved no prediction.

Every cell the derivation calls variant changed predictions in every seed. The smallest number of changed test predictions out of 1,000 in any seed was 129, 142, 205 and 119 for ridge without a bias (scale, translation, diagonal, dense); 98, 154 and 74 for ridge with a bias (scale, diagonal, dense); 21, 155 and 96 for $\ell_2$ logistic (scale, diagonal, dense); 293 and 146 for Euclidean 5-NN (diagonal, dense); 441, 374 and 181 for cosine 20-NN (translation, diagonal, dense); and 45 and 88 for standardize-then-ridge (rotation, dense). The scale by 10 only divides the effective $\lambda$ by 100, as derived above, yet it changed 98 to 183 predictions per seed for the two ridge probes and 21 to 50 for the logistic probe. The $\ell_2$ logistic cell under diagonal scaling mixes two effects: the printed count shows that 4 of its 5 fits stopped before reaching their tolerance, so its 189.0 changed predictions come from fits that had not reached the optimum, not from the non-invariance of the optimum alone. No other logistic fit stopped early.

Table A3 shows that the changes are not small perturbations. Per-feature scaling, which leaves the information in the features untouched, lowered the mean accuracy of Euclidean 5-NN from 0.719 to 0.570 and of cosine 20-NN from 0.769 to 0.518; in our reading, it lets a few stretched random features dominate the distances. It also lowered both penalized probes and the logistic probe, while least squares stayed at 0.800 and standardize-then-ridge at 0.824. Two printed means were not hypothesized. Ridge without a bias rose from 0.771 to 0.832 under translation; in our reading, a common offset in every feature vector lets the bias-free probe build an intercept from its weights. Whitened 5-NN, invariant to every map, had the lowest original accuracy of all eight protocols (0.590 [0.568–0.646]); in our reading, whitening gives the noisy low-variance directions of the random features as much weight in the distance as the informative ones. The protocols also rank differently: on the original features the logistic probe was most accurate and whitened 5-NN least, and per-feature scaling moved the cosine k-NN from sixth place to last.

Part B passed both exactness checks: the hat-matrix leave-one-out scores matched 100 refits to $1.1 \times 10^{-12}$, and $\text{PRESS}(10h, 100\lambda)$ matched $\text{PRESS}(h, \lambda)$ to a relative $9.1 \times 10^{-13}$. Re-selecting $\lambda$ on each feature set therefore picked exactly $10^{-2}$ instead of $10^{-4}$ after the scale by 10 in every seed and changed no prediction, and rotation and translation kept $10^{-4}$ and changed nothing, as derived. Re-selection did not rescue the other two maps: after diagonal scaling it chose $10^{-6}$ or $10^{-7}$ and still changed 75 to 164 predictions per seed (mean accuracy 0.799 against 0.831), and after the dense map 74 to 127 (mean accuracy 0.816). In our reading, tuning a probe’s hyperparameter per representation, as Kornblith et al. did, removes its dependence on the global scale of the features but not on their shape. The library cross-checks agreed: the hand-written least-squares and ridge probes matched scikit-learn to about $10^{-11}$ or better, and both k-NN variants gave identical predictions, including whitened 5-NN against scikit-learn’s Mahalanobis k-NN.

The example uses one fixed random encoder rather than learned features, so it says nothing about which self-supervised method is better; its point is the measuring instrument. For how many labels and which encoder change k-NN results, see the examples in k-nearest neighbors, which also shows k-NN’s sensitivity to features in incompatible units, and in representation learning, which compares learned and random representations under a linear probe and 5-NN.

Common Misconceptions

• “A linear probe measures the information in a representation.” Only an unregularized probe with a bias ignores invertible changes of coordinates. A probe with a penalty, which is the usual one, prefers representations in which the classes lie along large feature directions, so two encoders with the same information can score differently: in the example, ridge with a bias scored 0.831 on the original features and 0.737 on a per-feature rescaling of them (means over seeds). A linear probe also misses information that is present but not linearly decodable, although Kolesnikov et al. (2019) found a one-hidden-layer readout added little in their study.
• “k-NN evaluation has no hyperparameters.” It has fewer to tune than a linear or fine-tuned evaluation, which is why Caron et al. (2021) added it, but the metric, the normalization, $k$ and the weighting temperature are choices. Cosine and Euclidean k-NN ignore different transformations, and Wu et al. and DINO use different $k$.
• “Normalizing the features makes the comparison fair.” Standardization makes a protocol ignore per-feature scale and translation but makes it sensitive to rotations, which it otherwise ignored; whitening makes it ignore every invertible map but, in the example, cost accuracy. Each preprocessing chooses which differences between representations count.
• “A convex probe has no optimization issues.” Its optimum does not depend on initialization, but SGD schedules (Kolesnikov et al., 2019, §4.7) and solver tolerances still change the reported number; in the example, L-BFGS stopped early on badly scaled features.
• “Numbers from different papers can be compared.” Only under the same protocol, layer, augmentation, solver and hyperparameter search; Goyal et al. (2019) found that inconsistencies in these affected earlier comparisons and proposed a consistent evaluation setup.
• “Fine-tuning accuracy measures the representation.” Fine-tuning changes the representation, and it usually wins: in 179 of 192 dataset–model combinations in the study of Kornblith et al. (2019). Goyal et al. (2019) view self-supervised learning as a way to learn features rather than an initialization method, and therefore limit fine-tuning in their benchmark.

Where It Is Used

Linear classifiers on frozen features and fine-tuning are the standard protocols for self-supervised learning (Caron et al., 2021): methods such as contrastive learning, DINO and the masked autoencoder are compared by linear probes, k-NN or fine-tuning, and their tables are comparable only within one protocol. Transfer learning studies use the same protocols to decide between a frozen backbone and fine-tuning, and computer vision foundation models are adapted by zero-shot prompting, linear probing and fine-tuning, with the linear probe doubling as an evaluation protocol. The ViT paper used closed-form linear few-shot accuracy where full fine-tuning was too costly (ViT paper). Nearest-neighbour evaluation is close to image retrieval and embedding quality measures such as Recall@K, and low-shot probes connect to few-shot recognition. Domain models such as medical foundation models can be judged the same way, by frozen-feature probes against task-specific training.