Representation Learning

Representation learning, also called feature learning, is the part of machine learning that learns how to describe the input instead of having a person design the description. For images, a learned representation is the vector (or grid of vectors) that a trained network computes from the pixels, chosen so that later tasks such as image classification, object detection, retrieval or semantic segmentation become easy. Bengio, Courville and Vincent (2013) define it as learning representations of the data that make it easier to extract useful information when building classifiers or other predictors. This article is the overview of the subject: what a learned representation is, what makes one good, the main routes to one (supervised pretraining, self-supervised learning, and training against text), and how representations are evaluated and compared.

Definition

A representation of an image $x$ is the output $h = f(x)$ of a map $f$ from images to a feature space, usually $\mathbb{R}^d$. The map can be hand-designed, as with SIFT descriptors or a histogram of oriented gradients, or it can be learned from data, as when $f$ is the stack of layers of a neural network up to some layer and $h$ is that layer’s activation. Representation learning is the second case: the parameters of $f$ are fitted to data so that $h$ serves tasks the learner may not see during training. The learned part is usually called the encoder or backbone, and its output the features or, when compared by distance, an embedding.

Intuition

A raw pixel array is a representation too, but a poor one for most questions. Two photographs of the same cup from slightly different positions differ in almost every pixel; a cup and a bowl on the same table can differ in few. Questions that matter to a task (which object, where, in what pose) are entangled with questions that do not (lighting, small shifts, sensor noise). Bengio et al. (2013) put the difficulty this way: different representations can entangle and hide, more or less, the explanatory factors of variation behind the data.

A good representation is a change of coordinates that untangles them, so that a simple rule applied afterwards (a linear classifier, a distance threshold, a nearest-neighbor lookup) can answer the question. Deep networks build such coordinates in stages: early layers respond to local patterns such as edges and color blobs, and deeper layers to larger, more class-specific structures, as the feature map article shows with visualizations. Representation learning asks how to train those stages so that what they compute is useful beyond the one task they were trained on.

Formal Definition

Let $f_\theta : \mathcal{X} \to \mathbb{R}^d$ be an encoder with parameters $\theta$, learned on upstream (pretraining) data. A downstream task supplies labelled pairs $\{(x_i, y_i)\}_{i=1}^{n}$, and the representation is judged by how well a predictor built on $h = f_\theta(x)$ solves it.

Two properties are defined with respect to a set of transformations $\mathcal{T}$ of the image (shifts, crops, color changes, rotations). The encoder is invariant to $\mathcal{T}$ when

$$f_\theta\big(t(x)\big) = f_\theta(x) \quad \text{for all } t \in \mathcal{T},$$

and equivariant when the transformation acts predictably on the features, $f_\theta(t(x)) = \rho(t)\, f_\theta(x)$ for some map $\rho(t)$ on feature space; a convolutional layer’s feature maps, for instance, shift when the image shifts, up to boundary effects and striding. Invariance discards information about $t$; equivariance keeps it in a known form. Learned encoders are at best approximately invariant or equivariant.

The most common evaluation, the linear probe, freezes $\theta$ and trains only a linear classifier $W \in \mathbb{R}^{K \times d}$, $b \in \mathbb{R}^K$ on the downstream data:

$$\min_{W,\, b} \; \frac{1}{n} \sum_{i=1}^{n} \ell\big(W f_\theta(x_i) + b,\; y_i\big) + \lambda \lVert W \rVert_2^2 ,$$

with $\ell$ the cross-entropy loss of a softmax, which makes it multinomial logistic regression on fixed features. The problem is convex, so its result depends on the representation and the regularization weight $\lambda$, not on the initialization. A k-NN evaluation instead labels a test image by a vote among the $k$ training images whose features are closest to its own (see k-nearest neighbors), and fine-tuning minimizes the same loss over $\theta$ as well, starting from the pretrained values.

What Makes a Representation Good

There is no single training target for a “good” representation; Bengio et al. (2013, §3.6) single out the lack of a clear objective as what distinguishes representation learning from tasks such as classification. Their review instead lists general-purpose priors, assumptions about the world that a learner can build in. The ones that matter most for vision are below.

Multiple explanatory factors. Images are produced by many underlying factors (light sources, object shapes, materials, the camera), and what is learned about one factor should generalize across settings of the others. Representations in which several features can be active at once, which the review calls distributed, can distinguish exponentially many input configurations with a number of parameters that grows only linearly, while one-hot codes such as k-means cluster indices need a parameter set per region (§3.3).

Hierarchy and abstraction. Concepts are defined in terms of simpler concepts, so deep representations should become more abstract with depth, and more abstract features tend to be invariant to most local changes of the input (§3.4). Linear probes trained at many layers of Inception v3 and ResNet-50 support this for supervised networks: the linear separability of the features increased monotonically with depth (Alain & Bengio, 2016, in the 2018 revision). For self-supervised networks it depends on the architecture. With AlexNet, earlier work had found that representation quality degraded toward the last layers, but Kolesnikov, Zhai and Beyer (2019) found that in networks with skip connections the quality of the representation did not degrade toward the end of the model.

Invariance versus disentanglement. Invariant features are insensitive to variations that are uninformative for a task, but which variations those are depends on the task, and one representation is often used for many. Bengio et al. (2013, §3.5) therefore argue that the most robust approach is to disentangle as many factors as possible while discarding as little information about the data as is practical. Informally, a disentangled representation changes in one coordinate (or one group of coordinates) when one underlying factor changes; Locatello et al. (2019) note that no formal definition is widely accepted.

Shared factors across tasks. If many tasks depend on overlapping subsets of the same factors, a representation that captures those factors transfers: it can be learned once on one task, or without labels, and reused (§2, §3.1). This is the prior behind transfer learning and every route in the next section.

Other priors. The review also lists smoothness, the concentration of data near low-dimensional manifolds, natural clustering of classes, temporal and spatial coherence, sparsity, and simple (often linear) dependencies between high-level factors, the last of which is what a linear probe assumes.

Locatello et al. (2019) temper the disentanglement goal. They proved that unsupervised learning of disentangled representations is impossible without inductive biases on both the models and the data, and in a study of more than 12,000 models of six methods on seven datasets they could not identify well-disentangled models without supervision, nor did more disentanglement seem to make downstream learning need fewer samples.

Routes to a Learned Representation

From hand-designed features to learned ones

Before deep learning, vision pipelines computed hand-designed image features such as SIFT and HOG and pooled them, for instance into a bag of visual words, before a classifier; in such pipelines the only learned parts were the pooling vocabulary and the classifier. Bengio et al. (2013, §1) describe such feature engineering as important but labor-intensive. Their review dates a breakthrough in feature learning to 2006, initiated by Geoffrey Hinton and quickly followed by others: greedy layer-wise unsupervised pretraining, which learns a deep network one layer at a time, each layer without labels on the output of the layers below, and then uses the stack as features or as the initialization of a supervised network (§4). The autoencoder, which learns a code by reconstructing its input, was one of the building blocks. Principal component analysis, which the review calls probably the oldest feature extraction algorithm, is the linear special case: it learns essentially the same representation as a basic linear autoencoder, and because a composition of linear maps is linear, it cannot be stacked into a deeper representation (§5).

Supervised pretraining

AlexNet’s 2012 ImageNet result turned attention to the supervised route: an ensemble of its networks won the ILSVRC-2012 competition with a top-5 test error of 15.3%, against 26.2% for the second-best entry (Krizhevsky et al., 2012); Bengio et al. (2013, §2), recording the same result, give the previous figure as 26.1%. A convolutional neural network trained to classify ImageNet’s 1,000 classes turned out to produce features useful far beyond those classes. Donahue et al. (2014) took activations of such a network as fixed features for scene recognition, domain adaptation and fine-grained recognition, tasks that may differ significantly from the one the network was trained on, and reported results that significantly outperformed the state of the art on several benchmarks. Reusing an ImageNet-trained backbone, either frozen or fine-tuned, became a standard starting point; transfer learning covers how this is done and when it helps.

Supervised features encode what the labels reward. Class labels ask the network to become invariant to everything that does not change the class, which helps for tasks close to the source classes and can hurt for others. The digit example below shows this on a small scale.

Self-supervised learning

Self-supervised learning removes the labels by deriving a training signal from the images themselves; its article covers the methods and their history. In brief, it has gone through several families:

• Pretext tasks pose a hand-designed prediction problem. Doersch, Gupta and Efros (2015) trained a network to predict the relative position of two patches from one image, arguing that solving it requires recognizing objects and their parts; colorization, jigsaw puzzles and rotation prediction followed.
• Contrastive learning treats two augmented views of one image as a positive pair and views of other images as negatives. MoCo keeps the negatives in a queue encoded by a slowly updated (momentum) encoder (He et al., 2020); SimCLR draws them from a large batch and found that the composition of data augmentations plays a critical role (Chen et al., 2020). Its article, contrastive learning, covers the losses.
• Self-distillation trains a student network to match the outputs of a momentum teacher across crops of one image, without negatives. Caron et al. (2021) call their version, DINO, self-distillation with no labels; its features were strong for nearest-neighbor classification, as described in the evaluation section.
• Masked image modeling hides part of an image and predicts what is missing. The masked autoencoder masks a high proportion of patches, for example 75%, and reconstructs the missing pixels (He et al., 2022).

He et al. (2020) concluded from MoCo’s detection and segmentation results that the gap between unsupervised and supervised pretraining had been largely closed in many vision tasks. In each family the choice of augmentations, masks or pretext determines what the representation becomes invariant to, which plays the role that labels play in the supervised route.

Image–text pretraining

A third route supervises images with the text that accompanies them on the web. CLIP (Radford et al., 2021) trains an image encoder and a text encoder contrastively on 400 million image–text pairs so that matching pairs lie close together in a shared embedding space. Because the text side can embed any phrase, the text encoder turns the names or descriptions of new classes (“a photo of a dog.”) into the weights of a linear classifier, which allows zero-shot classification. In the authors’ comparison (arXiv version, Figure 6), zero-shot CLIP matched the average performance of a linear probe trained with four labelled examples per class on CLIP’s own features. Multimodal learning and vision-language pretraining cover this family.

Choosing among the routes

The routes differ in what supplies the training signal and therefore in what the representation keeps:

Route	Training signal	Typical invariances come from	Examples
Hand-designed	None (designed)	The designer	SIFT, HOG
Unsupervised reconstruction	Reconstructing the input	The bottleneck or noise	PCA, autoencoders
Supervised pretraining	Class labels	The label set	ImageNet classifiers
Self-supervised	The images themselves	Augmentations, masks, pretext	MoCo, SimCLR, DINO, MAE
Image–text	Paired captions	What captions mention	CLIP

The routes can also be combined: in the VTAB study described below, combining supervision with self-supervision was effective, and to a large extent self-supervision could replace labels (Zhai et al., 2019).

Evaluating Representations

Because no single objective defines a good representation, representations are compared by what they enable downstream. Four protocols are standard.

Linear probe. A linear classifier on frozen features, as in the formal definition. Its limited flexibility is the point: the arXiv version of the CLIP paper (§3.2) argues that fine-tuning adapts the representation to each dataset and can mask failures of pretraining, while linear classifiers expose them and need little hyperparameter tuning. Alain and Bengio (2016) named linear classifiers trained on intermediate layers “probes”; in their 2018 revision they liken them to thermometers, measuring instruments that never influence the model. Kolesnikov et al. (2019) warn that a linear model trained with stochastic gradient descent on fixed features is sensitive to the learning-rate schedule and may need many epochs to converge.

k-NN. A nearest-neighbor vote over stored features of the labelled training set. It needs no training, augmentation or hyperparameters beyond $k$, which is why Caron et al. (2021) report a weighted k-NN next to the linear probe. k-NN measures whether the features of same-class images are close, not merely linearly separable. The gap between the two varies: for DINO with a ViT-S, k-NN reached 74.5% top-1 on ImageNet against 77.0% for the linear probe, a closeness the authors found surprising and saw only with DINO on vision transformers, not with other self-supervised methods or with a ResNet-50.

Fine-tuning. Training the whole network on the downstream task, starting from the pretrained weights with a new head, sometimes with some layers frozen or with smaller learning rates for the pretrained layers. It is usually more accurate: in the study of Kornblith, Shlens and Le (2019) it beat logistic regression on fixed features in 179 of 192 dataset–model combinations. In our reading, it measures the pretrained weights as an initialization as much as a representation, which is the reason the CLIP authors give for preferring linear probes.

Transfer benchmarks. Suites of downstream tasks test generality rather than one dataset:

• Kornblith et al. (2019) compared 16 ImageNet architectures on 12 classification datasets. For networks trained with identical settings, ImageNet accuracy predicted transfer accuracy closely, but training choices such as label smoothing and dropout, which changed ImageNet accuracy by under 1%, made the fixed features much worse for transfer; the transfer learning article gives the numbers.
• The Visual Task Adaptation Benchmark (VTAB) defines a good representation as one that solves many diverse, unseen tasks with few labels (Zhai et al., 2019). It has 19 tasks in three groups: natural images (Caltech101, CIFAR-100, Pets and others), specialized remote-sensing and medical images, and structured tasks such as counting objects or predicting depth, orientation and position in rendered scenes. Its main setting, VTAB-1k, allows 1,000 labelled examples per task.
• Detection and segmentation transfer, with the encoder as the backbone of a detector on PASCAL VOC or COCO, is a common test for dense tasks; MoCo’s claim above rests on it.

Protocols disagree. In VTAB, supervised ImageNet pretraining gave excellent representations for natural images but was extremely limited on structured tasks, where self-supervised methods did slightly better, which the authors read as supervised models being invariant to features that structured understanding needs. Linear evaluation also changed conclusions (Zhai et al., 2019, §3.4). Averaged over the VTAB-1k tasks, the fully supervised ImageNet model scored 65.6% with fine-tuning but 57.1% with a linear probe (Table 9; the text gives 57.3%). The correlation between the two protocols was low on many specialized and structured tasks. A conditional BigGAN discriminator, at 42.5% under the linear probe (43.3% in the text), beat relative patch location and jigsaw (41.8% and 40.0%), yet after fine-tuning it scored 35.3% against their 50.8% and 51.1% (Table 6). The authors call linear evaluation a poor proxy for reduced sample complexity. Kolesnikov et al. (2019) found that the ranking of self-supervised methods also changed with the network architecture: a ResNet-50 v1 excelled with relative patch location but gave suboptimal results with rotation prediction. A representation’s quality is therefore a property of the encoder, the protocol and the target tasks together, and results compare only under matched conditions.

Examples

The script below compares five representations of scikit-learn’s $8 \times 8$ digit images. Each one is learned (when it is learned at all) on 540 images of the digits 0 to 4 and then evaluated with linear probes and 5-NN, both on held-out images of 0 to 4 and on the digits 5 to 9, which none of the encoders saw. The representations are the 64 raw pixels; PCA with 32 components; an untrained layer of 32 ReLU units with random Gaussian weights; the 32-unit hidden layer of an autoencoder trained to reconstruct the images; and the 32-unit hidden layer of a multilayer perceptron trained to classify 0 to 4. “5 shots” means the probe sees five labelled images per class, chosen at random. The seed controls the random weights, the network initializations and the five-shot samples; the loop runs seeds 0 to 4 and prints the mean and range. It runs in about three seconds with scikit-learn 1.7.2 and NumPy 2.3.5.

import numpy as np
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.neural_network import MLPClassifier, MLPRegressor

X, y = load_digits(return_X_y=True)
X = X / 16.0                                     # 1,797 8x8 images, pixels in [0, 1]
old, new = y < 5, y >= 5
# Learn representations on 60% of the images of digits 0-4 ...
Xs, Xo, ys, yo = train_test_split(X[old], y[old], train_size=0.6, stratify=y[old], random_state=0)
# ... and probe them on held-out 0-4 images and on the unseen digits 5-9.
Xo_tr, Xo_te, yo_tr, yo_te = train_test_split(Xo, yo, test_size=0.5, stratify=yo, random_state=0)
Xn_tr, Xn_te, yn_tr, yn_te = train_test_split(X[new], y[new], test_size=0.5, stratify=y[new], random_state=0)
D = 32                                           # size of every learned representation

def relu_layer(W, b):
    return lambda Z: np.maximum(Z @ W + b, 0.0)

def representations(seed):
    rng = np.random.default_rng(seed)
    pca = PCA(n_components=D, svd_solver="full").fit(Xs)
    W = rng.normal(0, 1 / np.sqrt(64), (64, D))
    ae = MLPRegressor(hidden_layer_sizes=(D,), max_iter=3000, random_state=seed).fit(Xs, Xs)
    sup = MLPClassifier(hidden_layer_sizes=(D,), max_iter=3000, random_state=seed).fit(Xs, ys)
    return {"raw pixels": lambda Z: Z,
            "PCA": pca.transform,                                    # unsupervised, linear
            "random ReLU": relu_layer(W, 0.0),                       # untrained
            "autoencoder": relu_layer(ae.coefs_[0], ae.intercepts_[0]),     # unsupervised
            "supervised": relu_layer(sup.coefs_[0], sup.intercepts_[0])}    # labels 0-4

def probe(f, X_tr, y_tr, X_te, y_te, seed, shots=None):
    if shots:                                    # keep `shots` labelled images per class
        rng = np.random.default_rng(seed)
        keep = np.concatenate([rng.choice(np.flatnonzero(y_tr == c), shots, replace=False)
                               for c in np.unique(y_tr)])
        X_tr, y_tr = X_tr[keep], y_tr[keep]
    F_tr, F_te = f(X_tr), f(X_te)
    lin = LogisticRegression(max_iter=5000).fit(F_tr, y_tr).score(F_te, y_te)
    knn = KNeighborsClassifier(n_neighbors=5).fit(F_tr, y_tr).score(F_te, y_te)
    return lin, knn

res = {}
for seed in range(5):
    for name, f in representations(seed).items():
        lin_o5, _ = probe(f, Xo_tr, yo_tr, Xo_te, yo_te, seed, shots=5)
        lin_n, knn_n = probe(f, Xn_tr, yn_tr, Xn_te, yn_te, seed)
        lin_n5, _ = probe(f, Xn_tr, yn_tr, Xn_te, yn_te, seed, shots=5)
        res.setdefault(name, []).append([lin_o5, lin_n, knn_n, lin_n5])

print(f"learn on {len(Xs)} images of 0-4; probe 0-4 on {len(Xo_tr)}/{len(Xo_te)}, 5-9 on {len(Xn_tr)}/{len(Xn_te)}")
print("accuracy in %, mean [min-max] over seeds 0-4")
print(f"{'':12s}{'0-4, 5 shots':>19s}{'5-9, linear':>19s}{'5-9, 5-NN':>19s}{'5-9, 5 shots':>19s}")
for name, r in res.items():
    r = 100 * np.array(r)
    print(f"{name:12s}" + "".join(f"{m:7.1f} [{lo:4.1f}-{hi:5.1f}]"
                               for m, lo, hi in zip(r.mean(0), r.min(0), r.max(0))))

learn on 540 images of 0-4; probe 0-4 on 180/181, 5-9 on 448/448
accuracy in %, mean [min-max] over seeds 0-4
                   0-4, 5 shots        5-9, linear          5-9, 5-NN       5-9, 5 shots
raw pixels     90.3 [85.6- 93.9]   97.5 [97.5- 97.5]   98.7 [98.7- 98.7]   90.4 [88.2- 93.3]
PCA            90.3 [86.2- 93.9]   97.8 [97.8- 97.8]   98.7 [98.7- 98.7]   90.3 [88.4- 93.8]
random ReLU    78.3 [69.1- 84.5]   90.9 [88.6- 93.1]   93.5 [90.8- 95.3]   76.9 [74.6- 79.9]
autoencoder    89.8 [85.1- 92.3]   93.8 [92.2- 95.3]   95.9 [94.9- 96.4]   83.6 [80.8- 86.8]
supervised     98.0 [95.0-100.0]   93.4 [92.2- 95.5]   91.6 [90.4- 94.4]   78.7 [75.0- 81.7]

The first column shows what labels buy. On the classes it was trained for, the supervised layer lets a linear probe reach 98.0% from five examples per class, about eight points above raw pixels; its 32 units have been shaped to separate exactly these classes. On digits it never saw, the same layer is the worst representation for k-NN, barely ahead of the random layer for five-shot probes, and below raw pixels in all three columns. Presumably it has become invariant to differences that did not matter for telling 0 to 4 apart but do matter for 5 to 9; in our reading this is a small-scale analogue of what Kornblith et al. found for ImageNet features on fine-grained datasets and VTAB on structured tasks.

PCA matches raw pixels almost exactly. Its 32 components project the images onto the directions of greatest variance, so distances, and with them the 5-NN results, hardly change; and a linear probe on a linear map of the pixels can express only classifiers that a probe on the pixels can also express, so the small differences come from regularization and from the discarded components. Dimensionality reduction alone does not make a representation better; it makes it smaller. The autoencoder, trained without labels, keeps more of what distinguishes the unseen digits than the supervised layer does (83.6% against 78.7% five-shot), but on this easy data it still does not beat the pixels. The random ReLU layer is the floor for an encoder of this size: in our interpretation, being untrained, it loses information without organizing what remains, and its in-domain five-shot results vary most from seed to seed.

The comparison also shows the protocols disagreeing in a small way. With all labels, both protocols put PCA and raw pixels first, but the supervised layer and the autoencoder are within half a point of each other under the linear probe, with overlapping ranges, while under 5-NN the autoencoder is ahead in every seed (its worst seed beats the supervised layer’s best). A linear probe needs only separability; k-NN needs same-class images to be close. Raw pixels are a much stronger baseline for small, centered digits than for photographs, so the ranking does not carry over to natural images; the point of the example is the evaluation, not the ranking.

Common Misconceptions

• “Better accuracy on the pretraining task means a better representation.” The pretraining task is a means. Kornblith et al. (2019) found regularizers that slightly raised ImageNet accuracy while making the penultimate-layer features much worse for transfer.
• “One protocol settles which representation is best.” Linear probes, k-NN and fine-tuning can rank representations differently, and VTAB found low correlation between linear and fine-tuned results on many non-natural tasks (Zhai et al., 2019).
• “More invariance is always better.” Invariance discards information. A representation invariant to orientation cannot report orientation, and VTAB found supervised ImageNet features extremely limited on structured tasks such as counting, depth and orientation.
• “Fewer dimensions means a more abstract representation.” Compression by PCA keeps most of the same information in fewer coordinates, and in the example changes almost nothing; in our reading, abstraction comes from what the training signal makes the encoder ignore.
• “Disentanglement can be learned from images alone.” Locatello et al. (2019) proved that unsupervised disentanglement is impossible without inductive biases on both models and data.

Where It Is Used

Learned representations are the input to almost every modern vision system. Classifiers, detectors and segmenters are built on a pretrained backbone and adapted by transfer learning and fine-tuning. Retrieval, deduplication and face recognition compare embeddings, often trained with metric learning. Self-supervised learning, with families such as contrastive learning, the masked autoencoder and DINO, provides encoders where labels are scarce, as in medical and satellite imagery. Image–text models such as CLIP provide encoders that can be queried with language, and computer vision foundation models extend all of these routes to larger data and models.