Deep Features

Deep features are the activations of a hidden layer of a trained neural network, read out and used as a description of the input image for some task other than the one the network was trained on. A convolutional network trained to classify ImageNet computes, on the way to its class scores, a sequence of intermediate representations; Donahue et al. (2014) took these activations from an AlexNet-style network, named them DeCAF (deep convolutional activation features), and showed that a linear classifier on them beat the hand-designed features of the time on object, scene, fine-grained and domain-adaptation benchmarks. Razavian et al. (2014) reached the same conclusion with the OverFeat network and called such features off-the-shelf. This article is about the features themselves: what changes from one layer to the next, which layer to read and how, and how the representations of different layers or networks can be compared. Using the features to solve a new task is the subject of transfer learning; turning them into vectors for retrieval, with normalization and whitening, is the subject of embeddings.

Definition

Let a network compute its output by composing layers, $f = \phi_L \circ \cdots \circ \phi_1$ . The deep feature of layer $l$ for an image $x$ is the activation

h_l(x) = \phi_l \circ \cdots \circ \phi_1 (x),

taken with all weights frozen, followed by a readout $r$ that turns it into a vector: flattening a convolutional feature map, pooling it over positions, or taking a fully connected layer’s output as it is. The feature is generic when one network’s $r(h_l(x))$ serves many tasks with only a simple model, such as a linear classifier or a distance, trained on top. The weights normally come from supervised training on a large labelled set, but nothing in the definition requires it: features of self-supervised networks, of image–text models, and even of networks with random or hand-set weights are deep features in the same sense, and how good they are is an empirical question.

Intuition

A convolutional network builds its representation in stages. Each stage filters the previous one with weights shared across positions, applies a nonlinearity and often pools over a neighbourhood. Three things change as the stages accumulate:

Composition. A unit of layer $l$ responds to a pattern of layer- $(l-1)$ responses, so units higher up respond to conjunctions of simpler patterns over larger receptive fields. The Zeiler and Fergus article describes what their deconvnet projections showed in an ImageNet network: corners and colour conjunctions in layer 2, textures in layer 3, class-specific parts in layer 4, whole objects in layer 5.
Invariance. Pooling and rectification discard where and in which phase a pattern occurred, so deeper features change less when the image moves a little. Exact invariance holds only for some transformations, as invariance and equivariance explains.
Specificity. Deeper features are shaped more by the labels the network was trained on. That helps when a new task resembles the training task and hurts when it does not.

The first two make deep features better descriptors than pixels; the third is why the best layer for a new task is often not the last one.

Formal Definition

Readouts. For a convolutional activation $h_l(x) \in \mathbb{R}^{C \times H \times W}$ , three common readouts are flattening ( $CHW$ numbers, which keep position), global average pooling,

r_{\text{avg}}(h)_c = \frac{1}{HW} \sum_{p} h_c(p),

which gives $C$ numbers and discards position, and second-order pooling of channel co-activations,

r_{\text{co}}(h)_{a,b,\mathbf{d}} = \frac{1}{|P_{\mathbf{d}}|} \sum_{p \in P_{\mathbf{d}}} h_a(p)\, h_b(p + \mathbf{d}),

over the positions $P_{\mathbf{d}}$ where both terms exist. With $\mathbf{d} = 0$ this is the Gram matrix that neural style transfer matches and the bilinear pooling used in fine-grained classification; with $\mathbf{d} \ne 0$ it also records which pattern tends to lie beside which.

Comparing representations. To ask how similar two layers, or the same layer in two networks, are, collect the readouts of the same $n$ images into matrices $X \in \mathbb{R}^{n \times p_1}$ and $Y \in \mathbb{R}^{n \times p_2}$ with centred columns. Kornblith, Norouzi, Lee and Hinton (2019) compare the two representations through their similarity structures: the entries of $XX^\top$ and $YY^\top$ are dot products between the examples, and

\langle \operatorname{vec}(XX^\top), \operatorname{vec}(YY^\top) \rangle = \operatorname{tr}(XX^\top YY^\top) = \lVert Y^\top X \rVert_F^2

(their Eq. 1). Normalizing gives linear centered kernel alignment,

\operatorname{CKA}(X, Y) = \frac{\lVert Y^\top X \rVert_F^2}{\lVert X^\top X \rVert_F \, \lVert Y^\top Y \rVert_F},

which lies between 0 and 1. It is the Hilbert–Schmidt independence criterion $\operatorname{HSIC}(K, L) = \operatorname{tr}(KHLH)/(n-1)^2$ with linear kernels $K = XX^\top$ , $L = YY^\top$ and centring matrix $H = I_n - \frac{1}{n}\mathbf{1}\mathbf{1}^\top$ , normalized as $\operatorname{HSIC}(K,L)/\sqrt{\operatorname{HSIC}(K,K)\operatorname{HSIC}(L,L)}$ (their Eqs. 3–4); other kernels, such as an RBF kernel, give other CKAs. For a linear kernel the authors note that CKA is equivalent to the RV coefficient and to Tucker’s congruence coefficient.

The older alternative is canonical correlation analysis (CCA), which finds pairs of directions in the two spaces whose projections are maximally correlated. With $Q_X, Q_Y$ orthonormal bases for the columns of $X$ and $Y$ , the canonical correlations $\rho_i$ are the singular values of $Q_X^\top Q_Y$ , and the mean squared canonical correlation is

R^2_{\text{CCA}} = \frac{\lVert Q_Y^\top Q_X \rVert_F^2}{p_1}

(their Eq. 7, with $p_1 \le p_2$ and matrices of full column rank; for rank-deficient matrices the example’s code divides by the smaller rank).

Properties

Which layer

Accuracy rises through the convolutional layers and can fall at the top. Donahue et al. took the activations of the $n$ th hidden layer as DeCAF $_n$ : DeCAF $_5$ after the last convolutional stage, DeCAF $_6$ and DeCAF $_7$ from the two fully connected layers. On Caltech-101 with 30 training images per class, a logistic regression reached 63.29% mean per-class accuracy on DeCAF $_5$ , 84.30% on DeCAF $_6$ and 84.87% on DeCAF $_7$ ; a linear SVM reached 77.12%, 84.77% and 83.24%, and the best combination by validation accuracy, an SVM on DeCAF $_6$ with dropout applied to the features, 86.91% (their Fig. 4, left). They did not evaluate layers below DeCAF $_5$ numerically, judging them unlikely to be semantically richer; in their t-SNE plots of the ILSVRC-2012 validation set, the first pooling layer showed no clear semantic clustering and DeCAF $_6$ a clear one (Fig. 1). Razavian et al. trained a linear SVM on every layer of OverFeat, counting convolution, pooling and rectification as separate layers, and plotted the mean average precision on PASCAL VOC 2007: it rose with depth except over the last two fully connected layers, with small dips at the rectification layers, which the authors attribute to half-rectification: it helps the network’s nonlinearity but loses information when a classifier reads that layer directly (their Fig. 2a). Their main results use the first fully connected layer. The Zeiler and Fergus article reports the same rise in their network’s Table 6, from 44.8% on Caltech-101 with layer-1 features, and a layer-5 SVM that slightly beat both classifiers on layer 7.

Specific features transfer less. As the Transfer Learning article summarizes Yosinski et al. (2014), copying the first $n$ layers of a network trained on one half of the ImageNet classes into a network for the other half worked almost perfectly for layers 1 and 2 and less well higher up, partly because upper layers were specific to their classes and partly because neighbouring middle layers had co-adapted. The best layer therefore depends on how far the new task is from the training task: for instance retrieval, the Embeddings article describes how Babenko et al. found the first fully connected layer better than the last hidden one.

The training data shape the features. Supervised features encode what the labels reward. The Places article reports that linear SVMs on the fc7 features of a network trained on scene images reached 54.32% on SUN397, against 42.61% with features of an ImageNet network, and a different 2019 study by Kornblith, Shlens and Le, “Do Better ImageNet Models Transfer Better?”, found, as Transfer Learning summarizes, that regularizers changing ImageNet accuracy by under 1% made the penultimate features much worse for linear probes.

How invariant

Pooling makes features insensitive to small shifts, but only exactly for some shifts. A stack of convolutions and pointwise nonlinearities is equivariant to integer shifts on a periodic grid; pooling at stride $s$ keeps that only for shifts that are multiples of $s$ ; a global average or a global co-occurrence over a periodic grid is then exactly invariant to them (the derivation is in invariance and equivariance; sampling and aliasing explains why other shifts leak through). In the arXiv versions of Zeiler and Fergus’s paper, as that article describes, small translations and scalings changed layer-1 features dramatically and layer-7 features less, and the network’s output was stable to translation and scaling but not to rotation. Invariance is a property of particular layers and readouts, not of deep features as such.

How similar layers are

What a similarity index should ignore. Kornblith et al. argue that an index for comparing representations should be invariant to orthogonal transformations and isotropic scaling, which preserve dot products and distances up to a factor, but not to arbitrary invertible linear maps (their §2). Their Table 1 lists linear CKA as invariant to the first two only, and CCA as invariant to all three. Invariance to invertible linear maps sounds desirable, since a following fully connected layer can undo such a map, but it has a cost. Their Theorem 1 states that if an index is invariant to invertible linear transformation of its first argument, then any two representations $X$ and $Y$ with $\operatorname{rank}(X) = \operatorname{rank}(Y) = n$ , the number of examples, receive the same similarity to any third. The authors note that some networks, especially convolutional ones, have more neurons in some layers than there are examples in the training set; CCA, which is invariant to invertible linear maps, cannot tell such layers apart.

Consistency across networks. As a sanity check, Kornblith et al. trained ten architecturally identical 10-layer CNNs from different initializations and asked, for each layer of one network, whether the most similar layer of another was the corresponding one (their Table 2). Linear CKA found it 99.3% of the time, RBF CKA with a bandwidth of 0.4 or 0.8 of the median distance 99.1% and 99.3%; mean squared CCA 10.6%, mean CCA 1.4%, SVCCA 15.1% and 9.9% for its two summaries, projection-weighted CCA 11.1% and linear regression 45.4%.

What CKA showed. In networks made 8 times deeper than their base CIFAR-10 network, CKA showed more than half the layers to be very similar to the last one, and logistic regression on each layer confirmed that accuracy stopped improving less than halfway through (their Fig. 3). In ResNets, post-residual activations were similar to one another and block interiors were not. Networks trained on CIFAR-10 and CIFAR-100 developed similar representations in their early layers, while similarity to untrained networks was much lower (Fig. 7). In the penultimate (global average pooling) layer of two networks, the shared subspace had about 10 dimensions, the number of CIFAR-10 classes (Fig. 8).

Training, random and hand-set weights

Generic features need training, but how much depends on the use. Donahue et al. report, in a footnote, that the same feature computed with randomized weights performed comparably to traditional hand-designed features. In perceptual similarity, the LPIPS article reports 64.3% agreement with human judgments for a randomly initialized network, against 63.2% for $\ell_2$ and 67.2% to 68.4% for self-supervised networks. The Perceptual Loss article describes how a VGG architecture with random weights failed at texture synthesis in Gatys et al.’s experiments. A fixed hierarchy with hand-set filters, such as a Gabor bank followed by pooled products, is the classical end of the same spectrum: texture analysis describes such statistics and the Portilla–Simoncelli model built from them.

Performance as descriptors

Donahue et al.’s results show how far a fixed feature and a linear classifier went in 2014. On the Office domain-adaptation benchmark, an SVM trained on Dslr images and tested on Webcam images reached 38.80% with SURF features and 91.48% with DeCAF $_6$ ; trained on both domains, 94.79% (their Table 1, which notes that with DeCAF the Dslr-to-Webcam shift is largely non-existent). On Caltech-UCSD Birds, logistic regression on DeCAF $_6$ reached 58.75%, and DeCAF $_6$ inside the deformable part descriptor pipeline 64.96%, against 50.98% for that pipeline with its original features (Table 2). On SUN-397 with 50 training images per class, Table 3 gives, for classifiers trained on dropped-out features as the text describes, 40.94% for logistic regression on DeCAF $_6$ (the entry in bold), 40.84% on DeCAF $_7$ , and 39.36% and 40.66% for the SVM, against 38.0% for Xiao et al.; the caption says that the bold result is logistic regression with DeCAF $_7$ , and the text that the best method by cross-validation was the SVM on DeCAF $_7$ at 40.94%, so the paper disagrees with itself about which combination gave that number. Razavian et al.’s results on classification and retrieval are summarized in transfer learning and embeddings.

Examples

The example builds a fixed two-layer hierarchy with hand-set filters, so that nothing is trained, and measures three things layer by layer: linear CKA between layers (with mean squared CCA beside it), sensitivity to shifts, and nearest-class-mean accuracy.

Layer 1 is eight complex Gabor filters from scikit-image’s gabor_kernel (frequencies 0.25 and 0.125 cycles per pixel, orientations 0°, 45°, 90° and 135°), rectified by taking the modulus of the response and average-pooled in $4 \times 4$ blocks: eight $16 \times 16$ maps (“L1 local”, 2,048 numbers). “L1 global” is the mean of each map, a global readout of layer 1 (8 numbers).
Layer 2 (“L2”) is the co-occurrence readout above, applied to the L1 maps: every channel pair at displacement $(0,0)$ and at one cell down, right, down-right and down-left (292 numbers).
Synthetic textures. Each $64 \times 64$ periodic image holds 12 junctions of one horizontal and one vertical bar ( $2 \times 10$ pixels), drawn additively, plus Gaussian noise of standard deviation 0.3. In class cross the bars cross at their middles, in ell they meet at their ends, in tee an end meets a middle (each in one of four rotations), and in apart the same bars lie independently. Every image contains the same 24 bars and the same ink, so the classes differ only in how bars are arranged relative to one another. The oracle row uses the generator’s own geometry (the offset from each horizontal bar’s centre to the nearest vertical bar’s centre) as a check that the labels can be recovered.
Photographs. 240 windows of $64 \times 64$ pixels, 40 from each of six images bundled with scikit-image (camera, astronaut, coffee, chelsea, rocket, coins, in grey). Each whole photograph is filtered once and the windows crop the responses, so no padding choice affects the window features.
Measures. CKA, $R^2_{\text{CCA}}$ and the shift measure use the 400 training textures of seed 0 and the 240 windows. Shift sensitivity is the RMS change of a representation under a horizontal shift by $s$ pixels, divided by the RMS distance of the unshifted features from their mean, so that 0 means invariant and about $\sqrt{2} \approx 1.41$ means as different as two unrelated images; textures are shifted circularly, photo windows are moved within the photograph. The co-occurrences wrap around the grid for the periodic textures and do not for the windows. Nearest-class-mean accuracy assigns each test texture to the class whose mean training feature is closest in Euclidean distance; whether to standardize each feature first is a choice with no single right answer, so both versions are printed. Seeds 0 to 4 each draw new training and test sets of 100 images per class.

The script runs in about 12 seconds on one CPU thread with NumPy 2.3.5, SciPy 1.16.3 and scikit-image 0.25.2; two runs printed identical output.

import os
os.environ["OMP_NUM_THREADS"] = "1"           # one BLAS thread: identical sums on every run
os.environ["OPENBLAS_NUM_THREADS"] = "1"
os.environ["MKL_NUM_THREADS"] = "1"
import numpy as np, scipy, skimage
from scipy import ndimage, signal
from skimage import data, color, img_as_float
from skimage.filters import gabor_kernel

SEEDS = (0, 1, 2, 3, 4)
N, POOL = 64, 4                               # image size, pooling block
KERNELS = [gabor_kernel(f, theta=t) for f in (0.25, 0.125)
           for t in (0, np.pi / 4, np.pi / 2, 3 * np.pi / 4)]      # 8 complex Gabor filters
SHIFTS = (0, 1), (1, 0), (1, 1), (1, -1)      # co-occurrence displacements, in pooled cells
print("numpy", np.__version__, "scipy", scipy.__version__, "scikit-image", skimage.__version__)

# ---------- synthetic junction textures: 12 junctions of two bars each, periodic 64 x 64 ----------
CLASSES = ["cross", "ell", "tee", "apart"]

def bars(kind, rng, n_junctions=12):
    """Bars as (orientation, row, col). H bar: 2 rows x 10 cols; V bar: 10 rows x 2 cols."""
    out = []
    for _ in range(n_junctions):
        y, x = rng.integers(0, N, 2)
        r = rng.integers(0, 4)                                    # one of four rotations
        if kind == "cross":                                       # bars cross at their middles
            out += [("H", y + 4, x), ("V", y, x + 4)]
        elif kind == "ell":                                       # bars meet at their ends
            out += [[("V", y, x), ("H", y + 8, x)], [("V", y, x + 8), ("H", y + 8, x)],
                    [("V", y, x), ("H", y, x)], [("V", y, x + 8), ("H", y, x)]][r]
        elif kind == "tee":                                       # an end meets a middle
            out += [[("H", y, x), ("V", y, x + 4)], [("H", y + 8, x), ("V", y, x + 4)],
                    [("V", y, x), ("H", y + 4, x)], [("V", y, x + 8), ("H", y + 4, x)]][r]
        else:                                                     # the same bars, placed independently
            y2, x2 = rng.integers(0, N, 2)
            out += [("H", y, x), ("V", y2, x2)]
    return out

def render(blist):
    img = np.zeros((N, N))
    for o, r, c in blist:
        rows = (r + np.arange(2 if o == "H" else 10)) % N
        cols = (c + np.arange(10 if o == "H" else 2)) % N
        img[np.ix_(rows, cols)] += 1.0                            # additive: same ink in every class
    return img

def oracle(blist):
    """Label check from the generator's geometry: offset from each H-bar centre to the nearest
    V-bar centre (periodic, L1), averaged: sum, |dy| and |dx|."""
    H = np.array([(r + 0.5, c + 4.5) for o, r, c in blist if o == "H"])
    V = np.array([(r + 4.5, c + 0.5) for o, r, c in blist if o == "V"])
    d = np.abs((H[:, None] - V[None] + N / 2) % N - N / 2)
    nn = d[np.arange(len(H)), d.sum(2).argmin(1)]
    return np.array([nn.sum(1).mean(), nn[:, 0].mean(), nn[:, 1].mean()])

def make(seed, n_per):
    rng = np.random.default_rng(seed)
    X, y, O, ink = [], [], [], []
    for ci, kind in enumerate(CLASSES):
        for _ in range(n_per):
            b = bars(kind, rng)
            clean = render(b)
            X.append(clean + rng.normal(0, 0.3, (N, N))); y.append(ci)
            O.append(oracle(b)); ink.append(clean.sum())
    return np.array(X), np.array(y), np.array(O), np.array(ink)

# ---------- the fixed hierarchy ----------
def kernel_fft(k):
    """FFT of kernel k on the periodic N x N grid, centre at pixel (0, 0)."""
    big, h = np.zeros((N, N), complex), k.shape[0] // 2
    for i in range(k.shape[0]):
        for j in range(k.shape[1]):
            big[(i - h) % N, (j - h) % N] += k[i, j]
    return np.fft.fft2(big)
KF = [kernel_fft(k) for k in KERNELS]

def energy_periodic(img):
    """Layer 1, before pooling: Gabor filtering (circular) and rectification by the modulus."""
    F = np.fft.fft2(img)
    return np.stack([np.abs(np.fft.ifft2(F * kf)) for kf in KF])          # (8, 64, 64)

def pool(E):
    c, h, w = E.shape
    return E.reshape(c, h // POOL, POOL, w // POOL, POOL).mean((2, 4))   # (8, 16, 16)

def cooc(A, wrap):
    """Layer 2: mean over positions p of A_a(p) * A_b(p + d), for every channel pair (a, b) and
    displacement d. wrap=True treats the 16 x 16 grid as periodic (exact for periodic textures);
    wrap=False averages only over pairs inside the grid (photo windows)."""
    C, h, w = A.shape
    feats = [(np.einsum("aij,bij->ab", A, A) / (h * w))[np.triu_indices(C)]]
    for dy, dx in SHIFTS:
        if wrap:
            B = np.roll(A, (-dy, -dx), axis=(1, 2))
            P = np.einsum("aij,bij->ab", A, B) / (h * w)
        else:
            x0, x1 = max(0, -dx), w - max(0, dx)
            P1, P2 = A[:, :h - dy, x0:x1], A[:, dy:, x0 + dx:x1 + dx]
            P = np.einsum("aij,bij->ab", P1, P2) / P1[0].size
        feats.append(P.ravel())
    return np.concatenate(feats)                                          # 36 + 4 * 64 = 292

def layers_from_maps(pix, A, wrap):
    return {"pixels": pix.ravel(), "L1 local": A.ravel(),
            "L1 global": A.mean((1, 2)), "L2": cooc(A, wrap)}

def texture_layers(img):
    return layers_from_maps(img, pool(energy_periodic(img)), wrap=True)

NAMES = list(texture_layers(np.zeros((N, N))))

def stack(feature_dicts):
    return {k: np.array([f[k] for f in feature_dicts]) for k in NAMES}

# ---------- measures ----------
def cka(X, Y):
    """Linear CKA (Kornblith et al., 2019): ||Y^T X||_F^2 / (||X^T X||_F ||Y^T Y||_F), centred columns."""
    X, Y = X - X.mean(0), Y - Y.mean(0)
    return np.linalg.norm(Y.T @ X) ** 2 / (np.linalg.norm(X.T @ X) * np.linalg.norm(Y.T @ Y))

def cka_gram(X, Y):
    """The same index from the n x n Gram matrices: HSIC(K, L) / sqrt(HSIC(K, K) HSIC(L, L))."""
    n = len(X); H = np.eye(n) - 1 / n
    hs = lambda K, L: np.trace(K @ H @ L @ H)
    K, L = X @ X.T, Y @ Y.T
    return hs(K, L) / np.sqrt(hs(K, K) * hs(L, L))

def r2_cca(X, Y, tol=1e-10):
    """Mean squared canonical correlation, from orthonormal bases of the centred columns."""
    def basis(Z):
        U, s, _ = np.linalg.svd(Z - Z.mean(0), full_matrices=False)
        return U[:, s > tol * s[0]]
    Qx, Qy = basis(X), basis(Y)
    rho = np.linalg.svd(Qx.T @ Qy, compute_uv=False)
    return (rho ** 2).sum() / min(Qx.shape[1], Qy.shape[1])

def shift_sensitivity(Z, Zs):
    """RMS change under the shift, relative to the RMS distance of the unshifted features from
    their mean: 0 means invariant, about 1.41 means as different as two unrelated images."""
    return np.sqrt(((Zs - Z) ** 2).sum(1).mean() / ((Z - Z.mean(0)) ** 2).sum(1).mean())

def ncm(Ztr, ytr, Zte, yte, standardize):
    if standardize:
        mu, sd = Ztr.mean(0), Ztr.std(0)
        sd[sd == 0] = 1.0
        Ztr, Zte = (Ztr - mu) / sd, (Zte - mu) / sd
    M = np.stack([Ztr[ytr == c].mean(0) for c in range(len(CLASSES))])
    d = ((Zte[:, None, :] - M[None]) ** 2).sum(2)
    return (d.argmin(1) == yte).mean()

def print_matrix(title, f, Z):
    print(title)
    print(" " * 10 + "".join(f"{k:>11s}" for k in NAMES))
    for a in NAMES:
        print(f"{a:10s}" + "".join(f"{f(Z[a], Z[b]):11.4f}" for b in NAMES))

# ================= A. data and exactness checks (textures, seed 0) =================
Xtr, ytr, Otr, ink = make(SEEDS[0], 100)
print(f"\nA. textures: {len(Xtr)} images, per class {np.bincount(ytr)}, "
      f"noise-free ink per image {ink.min():.0f} to {ink.max():.0f}")
E = energy_periodic(Xtr[0])
diff = max(np.abs(np.hypot(ndimage.convolve(Xtr[0], k.real, mode="wrap"),
                           ndimage.convolve(Xtr[0], k.imag, mode="wrap")) - e).max()
           for k, e in zip(KERNELS, E))
print(f"FFT filtering vs scipy.ndimage.convolve (wrap): max difference {diff:.1e}")
T = stack([texture_layers(x) for x in Xtr])
print("dimensions:", {k: v.shape[1] for k, v in T.items()})
rng = np.random.default_rng(123)
p = T["L2"].shape[1]
Q, _ = np.linalg.qr(rng.normal(size=(p, p)))                  # random rotation
A = np.diag(rng.uniform(0.1, 10, p))                          # random rescaling of each feature
print(f"CKA feature form vs Gram form: |difference| {abs(cka(T['L2'], T['L1 local']) - cka_gram(T['L2'], T['L1 local'])):.1e}")
print(f"CKA(X, XQ) - 1 = {cka(T['L2'], T['L2'] @ Q) - 1:.1e}   CKA(X, 3X) - 1 = {cka(T['L2'], 3 * T['L2']) - 1:.1e}")
print(f"rescaled features: CKA(X, XA) = {cka(T['L2'], T['L2'] @ A):.6f}   "
      f"R2_CCA(X, XA) = {r2_cca(T['L2'], T['L2'] @ A):.10f}")

# ================= B. similarity between layers =================
print_matrix("\nB1. linear CKA between layers, textures (n = 400)", cka, T)
print_matrix("B2. mean squared canonical correlation R2_CCA, textures", r2_cca, T)

def load(name):
    im = img_as_float(getattr(data, name)())
    return color.rgb2gray(im) if im.ndim == 3 else im
photos = {n: load(n) for n in ["camera", "astronaut", "coffee", "chelsea", "rocket", "coins"]}
SMAX, MARGIN = 16, max(k.shape[0] for k in KERNELS) // 2 + 1
energy_full = {n: np.stack([np.abs(signal.fftconvolve(p_, k, mode="same")) for k in KERNELS])
               for n, p_ in photos.items()}                    # filter each whole photograph once
rng = np.random.default_rng(0)
windows = [(n, rng.integers(MARGIN, p_.shape[0] - MARGIN - N + 1),
            rng.integers(MARGIN, p_.shape[1] - MARGIN - N - SMAX + 1))
           for n, p_ in photos.items() for _ in range(40)]

def window_layers(n, r, c):
    """Features of the 64 x 64 window at (r, c); its Gabor responses come from the whole photo,
    so they do not depend on how the window's border is padded."""
    return layers_from_maps(photos[n][r:r + N, c:c + N],
                            pool(energy_full[n][:, r:r + N, c:c + N]), wrap=False)

P = stack([window_layers(*w) for w in windows])
print_matrix(f"B3. linear CKA between layers, photo windows (n = {len(windows)})", cka, P)
print_matrix("B4. mean squared canonical correlation R2_CCA, photo windows", r2_cca, P)
print(f"R2_CCA(pixels, L1 local): textures {r2_cca(T['pixels'], T['L1 local']):.10f}, "
      f"photos {r2_cca(P['pixels'], P['L1 local']):.10f}")

# ================= C. shift sensitivity =================
S = (1, 2, 4, 8, 16)
print("\nC. shift sensitivity (0 = invariant, ~1.41 = unrelated image)")
print(f"{'':22s}" + "".join(f"{'s=' + str(s):>9s}" for s in S))
TS = {s: stack([texture_layers(np.roll(x, s, axis=1)) for x in Xtr]) for s in S}
PS = {s: stack([window_layers(n, r, c + s) for n, r, c in windows]) for s in S}
for label, Z0, ZS in (("textures, circular", T, TS), ("photos, window moved", P, PS)):
    for k in NAMES:
        print(f"{label[:8] + ' ' + k:22s}" + "".join(f"{shift_sensitivity(Z0[k], ZS[s][k]):9.4f}" for s in S))
exact = max(np.abs(TS[s][k] - T[k]).max() / np.abs(T[k]).max() for s in (4, 8, 16) for k in ("L1 global", "L2"))
print(f"textures, s = 4, 8, 16: largest relative change of L1 global and L2 {exact:.1e}")

# ================= D. nearest-class-mean accuracy (textures) =================
print(f"\nD. nearest-class-mean accuracy, textures, 100 train / 100 test per class, seeds {list(SEEDS)}")
acc = {(k, s): [] for k in NAMES + ["oracle"] for s in (False, True)}
for seed in SEEDS:
    Xa, ya, Oa, _ = make(seed, 100)
    Xb, yb, Ob, _ = make(1000 + seed, 100)
    Za, Zb = stack([texture_layers(x) for x in Xa]), stack([texture_layers(x) for x in Xb])
    Za["oracle"], Zb["oracle"] = Oa, Ob
    for k in NAMES + ["oracle"]:
        for st in (False, True):
            acc[(k, st)].append(ncm(Za[k], ya, Zb[k], yb, st))
print(f"{'':12s}{'raw: mean':>11s}{'range':>15s}{'standardized: mean':>21s}{'range':>15s}")
for k in NAMES + ["oracle"]:
    a, b = acc[(k, False)], acc[(k, True)]
    print(f"{k:12s}{np.mean(a):11.4f}{min(a):9.4f}-{max(a):.4f}{np.mean(b):21.4f}{min(b):9.4f}-{max(b):.4f}")

Output:

numpy 2.3.5 scipy 1.16.3 scikit-image 0.25.2

A. textures: 400 images, per class [100 100 100 100], noise-free ink per image 480 to 480
FFT filtering vs scipy.ndimage.convolve (wrap): max difference 5.0e-16
dimensions: {'pixels': 4096, 'L1 local': 2048, 'L1 global': 8, 'L2': 292}
CKA feature form vs Gram form: |difference| 8.3e-16
CKA(X, XQ) - 1 = 1.3e-15   CKA(X, 3X) - 1 = 2.2e-16
rescaled features: CKA(X, XA) = 0.988329   R2_CCA(X, XA) = 1.0000000000

B1. linear CKA between layers, textures (n = 400)
               pixels   L1 local  L1 global         L2
pixels         1.0000     0.7238     0.0877     0.0935
L1 local       0.7238     1.0000     0.1044     0.0959
L1 global      0.0877     0.1044     1.0000     0.7688
L2             0.0935     0.0959     0.7688     1.0000
B2. mean squared canonical correlation R2_CCA, textures
               pixels   L1 local  L1 global         L2
pixels         1.0000     1.0000     1.0000     1.0000
L1 local       1.0000     1.0000     1.0000     1.0000
L1 global      1.0000     1.0000     1.0000     0.9921
L2             1.0000     1.0000     0.9921     1.0000
B3. linear CKA between layers, photo windows (n = 240)
               pixels   L1 local  L1 global         L2
pixels         1.0000     0.0769     0.0254     0.0175
L1 local       0.0769     1.0000     0.7972     0.7072
L1 global      0.0254     0.7972     1.0000     0.8414
L2             0.0175     0.7072     0.8414     1.0000
B4. mean squared canonical correlation R2_CCA, photo windows
               pixels   L1 local  L1 global         L2
pixels         1.0000     1.0000     1.0000     1.0000
L1 local       1.0000     1.0000     1.0000     1.0000
L1 global      1.0000     1.0000     1.0000     1.0000
L2             1.0000     1.0000     1.0000     1.0000
R2_CCA(pixels, L1 local): textures 1.0000000000, photos 1.0000000000

C. shift sensitivity (0 = invariant, ~1.41 = unrelated image)
                            s=1      s=2      s=4      s=8     s=16
textures pixels          1.0697   1.2160   1.2571   1.3684   1.4187
textures L1 local        0.2490   0.4840   0.8701   1.2697   1.4251
textures L1 global       0.0000   0.0000   0.0000   0.0000   0.0000
textures L2              0.0178   0.0249   0.0000   0.0000   0.0000
photos,  pixels          0.2667   0.3908   0.5158   0.6632   0.8394
photos,  L1 local        0.1612   0.3156   0.5810   0.9123   1.1539
photos,  L1 global       0.0294   0.0585   0.1149   0.2209   0.4003
photos,  L2              0.0402   0.0801   0.1559   0.2880   0.5060
textures, s = 4, 8, 16: largest relative change of L1 global and L2 3.7e-16

D. nearest-class-mean accuracy, textures, 100 train / 100 test per class, seeds [0, 1, 2, 3, 4]
              raw: mean          range   standardized: mean          range
pixels           0.2435   0.2150-0.2700               0.2395   0.2175-0.2625
L1 local         0.3735   0.3450-0.4075               0.3990   0.3800-0.4175
L1 global        0.6745   0.6500-0.6900               0.7225   0.7000-0.7525
L2               0.8085   0.7650-0.8275               0.8330   0.8200-0.8500
oracle           0.9700   0.9625-0.9775               0.9695   0.9600-0.9775

Checks. The FFT filtering agrees with direct convolution, the two forms of CKA agree, and CKA is unchanged, to rounding error, by a random rotation of the L2 features and by multiplying them by 3. Rescaling each feature by its own factor between 0.1 and 10 lowers CKA to 0.988329 but leaves $R^2_{\text{CCA}}$ at 1, the difference between the two indexes in Kornblith et al.‘s Table 1. The oracle reaches 96.0% to 97.75%, so the labels can be recovered from the images’ arrangement of bars.

Similarity between layers (B). On the textures, layer 2 is far from the pixels (CKA 0.0935) and almost as far from layer 1’s local maps (0.0959), but close to layer 1’s global means (0.7688): a co-occurrence at displacement zero includes each channel’s mean squared energy. The plan’s prediction that CKA would fall with distance from the pixels held on both data sets: pixels against L1 local exceeds pixels against L2, and L1 local against L2 exceeds pixels against L2. On the textures the second comparison holds by only 0.0024 in the printed values, and CKA is computed on the seed-0 images alone, so the example cannot say whether another draw would reverse it. The same pair, pixels against L1 local, gives 0.7238 on the textures and 0.0769 on the photographs. In our reading, the textures share no layout, so in both representations each image is nearly unrelated to every other and both similarity matrices are dominated by their diagonals, which CKA then rates alike; the photographs share brightness and layout, which pixels and band-pass energies weight very differently. A CKA value describes two representations on particular data, not the two layers in general.

Almost every $R^2_{\text{CCA}}$ entry is 1. This is the saturation behind Theorem 1, in the form it takes for centred data: the comparison is blind once a representation has as many independent directions as the data allow. Theorem 1 assumes rank $n$ , which centred data cannot reach. By our derivation from the definition, centred columns can span at most the $n - 1$ directions orthogonal to the constant vector, and a representation that spans all of them contains every direction of any other representation of the same images, so all canonical correlations with it equal 1. With 400 textures, the pixels (4,096 columns) and L1 local (2,048) are wide enough to reach that span, and the line with ten decimals shows $R^2_{\text{CCA}} = 1$ between them on both data sets, as the plan predicted, while their CKA stays well below 1; the only pair below 1 is L1 global against L2 (0.9921), both narrower than 399. With 240 windows every entry prints as 1.0000, L2’s 292 columns included. CKA still separates the same pairs.

Shift sensitivity (C). On the periodic textures, the global means of layer 1 do not change at all under any circular shift, since a global average of circularly filtered maps cannot see where a pattern lies. Layer 2 is exactly invariant (largest change $3.7 \times 10^{-16}$ ) for shifts of 4, 8 and 16 pixels, multiples of the pooling stride, and is not for 1 and 2 pixels (0.0178 and 0.0249), because the $4 \times 4$ blocks then cut the image differently. L1 local is less sensitive than pixels at small shifts and becomes as sensitive as pixels by 16 pixels, four cells; in our reading, a shift longer than the 10-pixel bars leaves the shifted maps as unrelated to the originals as the shifted pixels are. The plan’s predictions for the textures held: at 1 and 2 pixels, L1 local is below the pixels and L2 below L1 local.

For the photographs, the plan predicted that at every shift L2 would be below L1 local, L1 local below the pixels, and L1 global below L1 local. That prediction failed at shifts of 4, 8 and 16 pixels, where L1 local is more sensitive than the pixels (0.5810 against 0.5158, 0.9123 against 0.6632, 1.1539 against 0.8394); it held at 1 and 2 pixels, and L2 and L1 global stayed below L1 local at every shift. In our reading, the measure divides by the spread of each representation across the 240 windows: for pixels that spread includes the brightness differences between the six photographs, which a shift within one photograph keeps, while the Gabor filters, which respond little to uniform brightness, leave L1 local’s spread to the local structure that a shift moves. Neither L2 nor L1 global is invariant on the photographs, because content enters and leaves a window that is not periodic; their sensitivity at 16 pixels (0.5060 and 0.4003) is still under half that of L1 local.

Nearest class mean (D). Over both versions of the classifier, pixels are at chance (21.5% to 27.0%, against 25%) and L1 local maps not far above it (34.5% to 41.75%), because the junctions lie at random positions. L1 global already reaches 65.0% to 75.25% although every class has the same bars and ink: in our reading, junctions change the local filter energies, where overlapping bars add up and where bar ends meet. L2 reaches 76.5% to 85.0%. The plan predicted that L2’s lowest seed would lie above the highest seed of each other layer under both versions of the classifier, and it held: 0.7650 against at most 0.6900 raw, 0.8200 against at most 0.7525 standardized. The gap to the oracle shows how much of the arrangement this hand-set second layer still misses. Two published examples are related. Fine-Grained Image Classification compares global average with bilinear pooling of the same eight-filter Gabor bank for a small cue on a synthetic bird’s head. Texture Analysis shows mean Gabor energies at chance on textures with equal spectra, where a higher-order statistic of the same filters separates them. Here the point is how one representation changes from layer to layer under three measures at once.

Common Misconceptions

“The last hidden layer gives the best features.” In the studies above, accuracy rose through the convolutional layers, but the top can be worse for a new task: DeCAF $_7$ was below DeCAF $_6$ for the SVM on Caltech-101, OverFeat’s last two layers did not improve VOC classification, and in Zeiler and Fergus’s Table 6 a layer-5 SVM slightly beat layer 7 on Caltech-101. The best layer depends on how close the new task is to the training task.
“Deep features are invariant to translation.” Convolutions are equivariant, and invariance comes only from pooling, exactly only for some shifts on a periodic grid. In the example, a layer-2 readout is exact for multiples of the pooling stride, not for one- or two-pixel shifts, and on photograph windows it changes with every shift. The pooling article measures the same effect for one-pixel shifts.
“High CCA means two layers compute the same thing.” Any index invariant to invertible linear maps gives the same answer for all representations of full rank (Kornblith et al.’s Theorem 1); with wide layers and few images, CCA can equal 1 for representations that CKA rates as dissimilar, as in the example. CKA does not have this limitation.
“Layer similarity is a property of the network.” CKA and CCA compare representations on a set of inputs; the same two layers of the example score 0.72 on one set of images and 0.08 on another.
“Only trained features are useful.” Random and hand-set features are weaker than trained ones for semantic tasks, but not worthless: DeCAF’s randomized-weight version matched hand-designed features, and in the LPIPS study a random network slightly beat $\ell_2$ . Pooled responses of fixed filter banks are the traditional texture statistics, as texture analysis describes.

Where It Is Used

Transfer to new tasks. A linear classifier on frozen features is the simplest use; transfer learning compares it with fine-tuning, and representation learning uses it as an evaluation protocol for self-supervised networks, and evaluating representations describes such probes and how their settings move the results.
Retrieval and matching. Pooled and normalized deep features serve as image embeddings for image retrieval.
Detection and segmentation. R-CNN classified region proposals with the features of an ImageNet-pretrained network, fine-tuned for detection; later detectors read several layers of a backbone network, as in the feature pyramid network.
Perceptual comparison and synthesis. LPIPS and the perceptual loss compare images by their deep features; neural style transfer matches their Gram statistics.
Scene and texture recognition. Scene recognition compares features of networks trained on objects and on scenes, and texture analysis describes convolutional features pooled over the image without regard to position.
Understanding networks. Feature visualization and neural network interpretability ask what single units respond to; similarity indexes such as CKA compare whole layers within and across networks.