Self-Supervised Learning

Self-supervised learning trains a network on unlabeled images by posing a prediction problem whose answers come from the images themselves: how an image was rotated, which two crops came from the same photograph, which pixels lie under a mask. The answers are not the goal. The goal is the representation learned along the way, which is then reused for image classification, object detection, semantic segmentation and other labelled tasks. This article is the overview; contrastive learning and the masked autoencoder have their own articles.

Purpose

Supervised pre-training on ImageNet needs a human label per image. Gidaris et al. (2018) cite that cost and the abundance of unlabeled images. Self-supervised learning replaces the label with a pretext task, a prediction problem built automatically from the data; Doersch et al. (2015) put “pretext” in quotation marks because the task is only a means to learn useful features. The aim is a representation that transfers: on top of it, a small labelled set suffices for a linear classifier or for fine-tuning.

When to Use It

It pays off when unlabeled images are plentiful and labels scarce or expensive (medical scans, satellite images, inspection), and when one backbone must serve many tasks, the use DINOv2 was designed for (Oquab et al., 2024). It is a weaker choice when a large labelled set exists for the exact target task, or when compute is tight: the methods below train for hundreds of epochs on ImageNet-scale data, 1,600 for MAE’s best models. In our reasoning, it also fails when the pretext task or augmentations remove what the target task needs; an encoder trained to ignore color will be poor at telling ripe fruit from unripe.

How It Works

Every method turns an unlabeled image $x$ into an input and a target computed from $x$, trains an encoder $f_\theta$ (a convolutional network or a transformer) with an ordinary loss, often a cross-entropy loss, and then discards the pretext-specific parts and keeps the representation $h = f_\theta(x)$.

Two-view notation. Most modern methods draw two random augmentations $t, t' \sim \mathcal{T}$ (crops, flips, color changes, blur) and form views $v = t(x)$ and $v' = t'(x)$. The encoder gives $h = f_\theta(v)$, a small projection head $g$ gives the embedding $z = g(h)$, and $\mathrm{sim}(u, w) = u^\top w / (\lVert u \rVert \lVert w \rVert)$ is the cosine similarity. Training makes $z$ and $z'$ agree; afterwards the head is dropped. Chen et al. (2020) found that linear evaluation of $h$ was more than 10% better with a nonlinear head than with none.

Collapse. Agreement alone has a trivial solution: an encoder that maps every image to the same vector. Two-view families differ mainly in how they rule it out:

• Negatives. Contrastive methods also push different images apart. SimCLR’s NT-Xent loss for view $i$ with positive partner $j$, among the $2N$ views of $N$ images, is

$$\ell_{i,j} = -\log \frac{\exp\big(\mathrm{sim}(z_i, z_j)/\tau\big)}{\sum_{k=1}^{2N} \mathbb{1}[k \neq i]\, \exp\big(\mathrm{sim}(z_i, z_k)/\tau\big)},$$

a softmax cross-entropy, with temperature $\tau$, that picks the partner out of the other views. A constant encoder makes the negatives as similar as the positive and cannot lower it.

• Predictor and stop-gradient. SimSiam passes each embedding through a predictor MLP $q$, $p = q(z)$, and minimizes $\tfrac12 D(p, \mathrm{sg}(z')) + \tfrac12 D(p', \mathrm{sg}(z))$, with $D$ the negative cosine similarity and $\mathrm{sg}$ the stop-gradient. Without the stop-gradient it collapsed to 0.1% linear accuracy, and without the predictor it also failed (Chen & He, 2021). BYOL’s target is a moving average of the online network; Grill et al. (2020) hypothesized that the predictor and the slow target together prevent collapse.
• Centering and sharpening. DINO centers the momentum teacher’s outputs, which stops one dimension dominating but alone pushes toward a uniform output, and sharpens them with a low temperature, which has the opposite effect; together they avoid collapse (Caron et al., 2021).
• Variance and decorrelation. Barlow Twins drives the cross-correlation matrix of the two embeddings toward the identity (Zbontar et al., 2021). VICReg adds a hinge loss keeping each embedding dimension’s standard deviation over the batch above a threshold, plus a decorrelation term (Bardes et al., 2022).
• Balanced clusters. DeepCluster reassigns empty clusters and samples images uniformly over pseudo-labels (Caron et al., 2018); SwAV computes balanced soft assignments to learned prototypes with the Sinkhorn–Knopp algorithm (Caron et al., 2020).

Reconstruction targets (pixels in inpainting and MAE, fixed tokens in BEiT) cannot collapse, because they do not come from the network being trained; iBOT’s targets come from an online teacher, so it keeps DINO’s centering and sharpening (Zhou et al., 2022).

Augmentations are the inductive bias. For two-view methods, the augmentations decide what the representation ignores. Chen et al. (2020) found no single transformation sufficient; random cropping plus color distortion stood out, which the authors attribute to crops of one image sharing a color distribution that the network can otherwise match on. Doersch et al. (2015) met a similar shortcut: their network located patches by chromatic aberration (in some lenses one color channel is shrunk toward the image center relative to the others) until preprocessing removed it. With random crops only, BYOL’s ImageNet linear accuracy (300 epochs) fell by 13.1 points to 59.4%, and SimCLR’s (in BYOL’s reimplementation) by 27.6 points to 40.3% (Grill et al., 2020). Masked modeling needs far less: MAE worked with crops alone and decently with no augmentation (He et al., 2022).

Variants

Pretext tasks (2015–2018)

The first wave designed hand-crafted problems for convolutional networks. Context prediction asks which of eight neighboring positions a second patch came from, with a gap and jitter between patches (Doersch et al., 2015). Jigsaw puzzles ask which of 1,000 permutations of nine tiles, chosen out of $9! = 362{,}880$ for large Hamming distance, was applied (Noroozi & Favaro, 2016). Colorization predicts the ab channels of a CIE Lab image from its lightness as classification over 313 quantized colors (Zhang et al., 2016). Context encoders inpaint a missing region with a reconstruction loss and an optional adversarial loss (Pathak et al., 2016). Rotation prediction classifies which of 0°, 90°, 180° or 270° was applied; its AlexNet reached 54.4% mAP on PASCAL VOC 2007 detection, 2.4 points below supervised pre-training (Gidaris et al., 2018). A shared weakness is that the last layers specialize to the pretext: on CIFAR-10, a non-linear classifier on the final block of a five-block rotation network reached 50.4%, against 89.8% on its second block (Gidaris et al., 2018, Table 1).

Contrastive instance discrimination (2018–2020)

Wu et al. (2018) treated every image as its own class, using noise-contrastive estimation and a memory bank of stored features. Contrastive Predictive Coding named the loss InfoNCE; on images it predicts the latent codes of lower rows of patches from the rows above (van den Oord et al., 2018). MoCo keeps negatives in a queue encoded by a momentum encoder, decoupling their number from the batch size (He et al., 2020); SimCLR takes them from a large batch and adds the projection head (Chen et al., 2020); MoCo v2 adopted SimCLR’s MLP head and stronger augmentation (Chen, Fan, Girshick & He, 2020).

Non-contrastive, self-distillation and clustering (2018–2021)

BYOL and SimSiam drop negatives. DINO calls its version self-distillation with no labels: a student matches the softmax output of a momentum teacher across crops of different sizes (multi-crop). With Vision Transformers, the last-layer attention of its class token highlighted objects without supervision (Caron et al., 2021). DeepCluster alternates k-means on the current features with training to predict the cluster indices (Caron et al., 2018); SwAV predicts one view’s cluster code from the other view, online (Caron et al., 2020).

Masked image modeling (2021–)

BEiT masks about 40% of a Vision Transformer’s patches and predicts discrete visual tokens from a pretrained discrete VAE with 8,192 tokens (Bao et al., 2022). The masked autoencoder (MAE) masks 75% of patches, encodes only the visible ones and reconstructs pixels with a light decoder, which sped up training by 3× or more (He et al., 2022). iBOT replaces the fixed tokenizer with an online teacher, combining masked prediction with DINO-style self-distillation (Zhou et al., 2022). DINOv2 combines the DINO and iBOT losses, trains on LVD-142M, a curated set of 142 million images, and distills a 1-billion-parameter ViT into smaller models (Oquab et al., 2024).

Family	Method	Year	Training signal	Avoids collapse by	Article
Pretext task	Context prediction	2015	Relative position of two patches	Not applicable (classification)	This article
Pretext task	Jigsaw	2016	Permutation of nine tiles	Not applicable	This article
Pretext task	Colorization	2016	ab color from lightness	Not applicable	This article
Pretext task	Rotation (RotNet)	2018	Which of four rotations	Not applicable	This article
Contrastive	Instance discrimination	2018	Each image is a class	Negatives from a memory bank	Contrastive learning
Contrastive	MoCo	2020	Two views agree	Negatives from a queue	Contrastive learning
Contrastive	SimCLR	2020	Two views agree	Negatives from the batch	Contrastive learning
Non-contrastive	BYOL, SimSiam	2020, 2021	Predict the other view’s embedding	Predictor and stop-gradient	This article
Self-distillation	DINO	2021	Match the teacher’s output	Centering and sharpening	This article
Redundancy reduction	Barlow Twins, VICReg	2021, 2022	Two views agree	Decorrelation and variance terms	This article
Clustering	DeepCluster, SwAV	2018, 2020	Predict cluster assignments	Balanced clusters	This article
Masked modeling	BEiT, MAE, iBOT	2022	Masked patches	Fixed target (pixels, tokens); iBOT uses centering	Masked autoencoder

How to Apply It

Representations are compared with standard protocols, usually on ImageNet:

• Linear probing: a linear classifier on frozen features, trained on the full labelled training set, reporting top-1 validation accuracy.
• k-NN: a vote among the nearest training features; DINO uses Wu et al.’s weighted k-NN with 20 neighbors, which needs no tuning or augmentation (Caron et al., 2021).
• Fine-tuning: training the whole network, sometimes on 1% or 10% of the labels.
• Transfer: the encoder as a detection or segmentation backbone on PASCAL VOC, COCO or ADE20K, against supervised pre-training.

Protocols can disagree. After 1,600 pre-training epochs, MAE’s ViT-L scored 75.1% linear but 85.1% fine-tuned (He et al., 2022, Figure 7), and the authors note, citing earlier work, that linear probing correlates poorly with transfer to detection. Judge a method by the protocol closest to its intended use.

The script below runs the whole recipe on synthetic $16 \times 16$ images of upright T, L and chevron shapes at random positions, sizes and stroke widths, with noise. It discards the shape labels of 2,000 images, builds 8,000 rotated copies labelled by their rotation, and trains a small network on that pretext task: 16 convolutional filters ($5 \times 5$), ReLU and global max pooling, with a hand-written gradient checked against finite differences. Then, with five labelled images per shape, it trains a linear classifier on raw pixels, on the untrained network’s features, and on the rotation-pretrained features. It runs in about three seconds with NumPy 2.3.

import numpy as np
from numpy.lib.stride_tricks import sliding_window_view

S, K = 16, 5                                   # image size, filter size
yy, xx = np.mgrid[0:S, 0:S].astype(float)

def shape(kind, rng):
    """An upright T, L or chevron at a random position, size and thickness, plus noise."""
    cy, cx = 7.5 + rng.uniform(-3, 3, size=2)
    h, t = rng.uniform(4, 6), rng.uniform(0.9, 1.6)
    y, x = yy - cy, xx - cx
    if kind == 0:    # T
        m = ((abs(y + h) < t) & (abs(x) < h)) | ((abs(x) < t) & (abs(y) < h))
    elif kind == 1:  # L
        m = ((abs(x + h) < t) & (abs(y) < h)) | ((abs(y - h) < t) & (abs(x) < h))
    else:            # chevron pointing up
        m = (abs(abs(x) - (y + h) / 2) < t) & (abs(y) < h)
    return m + 0.3 * rng.standard_normal((S, S))

def make(n, rng):
    k = rng.integers(0, 3, n)
    return np.stack([shape(c, rng) for c in k]), k

def rotations(imgs):
    """Pretext data: each image at 0, 90, 180 and 270 degrees; the label is the rotation."""
    X = np.concatenate([np.rot90(imgs, r, axes=(1, 2)) for r in range(4)])
    return X, np.repeat(np.arange(4), len(imgs))

def patches(imgs):                             # (n, 144 positions, 25)
    return sliding_window_view(imgs, (K, K), axis=(1, 2)).reshape(len(imgs), -1, K * K)

def features(p, imgs):
    """16 filters of 5x5, ReLU, then global max pooling over the 12x12 positions."""
    A = np.maximum(patches(imgs) @ p[0] + p[1], 0)
    return A.max(1), A.argmax(1)

def loss_grad(p, imgs, y):
    Wc, bc, W, b = p
    h, idx = features(p, imgs)
    Z = h @ W + b
    P = np.exp(Z - Z.max(1, keepdims=True)); P /= P.sum(1, keepdims=True)
    n = len(y)
    loss = -np.log(P[np.arange(n), y]).mean()
    dZ = P; dZ[np.arange(n), y] -= 1; dZ /= n
    dh = (dZ @ W.T) * (h > 0)                  # only the winning position gets gradient
    win = patches(imgs)[np.arange(n)[:, None], idx]      # (n, 16, 25)
    return loss, [np.einsum('nfk,nf->kf', win, dh), dh.sum(0), h.T @ dZ, dZ.sum(0)]

def linear_probe(Ftr, ytr, Fte, yte, steps=500, lr=0.5, lam=1e-3):
    """Softmax regression on frozen features; returns test accuracy."""
    mu, sd = Ftr.mean(0), Ftr.std(0) + 1e-6
    Ftr, Fte = (Ftr - mu) / sd, (Fte - mu) / sd
    W, b = np.zeros((Ftr.shape[1], 3)), np.zeros(3)
    for _ in range(steps):
        Z = Ftr @ W + b
        P = np.exp(Z - Z.max(1, keepdims=True)); P /= P.sum(1, keepdims=True)
        P[np.arange(len(ytr)), ytr] -= 1; P /= len(ytr)
        W -= lr * (Ftr.T @ P + lam * W); b -= lr * P.sum(0)
    return ((Fte @ W + b).argmax(1) == yte).mean()

def run(seed, check=False):
    rng = np.random.default_rng(seed)
    Xr, yr = rotations(make(2000, rng)[0])     # shape labels discarded: 8,000 pretext images
    Xr_test, yr_test = rotations(make(500, rng)[0])
    p = [rng.standard_normal((K * K, 16)) * np.sqrt(2 / (K * K)), np.zeros(16),
         rng.standard_normal((16, 4)) * 0.25, np.zeros(4)]
    p_random = [q.copy() for q in p]           # control: the same network, untrained

    if check:                                  # finite-difference gradient check
        _, g = loss_grad(p, Xr[:20], yr[:20])
        errs = []
        for k in range(4):
            for _ in range(5):
                i = tuple(rng.integers(0, s) for s in p[k].shape)
                old = p[k][i]
                p[k][i] = old + 1e-6; lp = loss_grad(p, Xr[:20], yr[:20])[0]
                p[k][i] = old - 1e-6; lm = loss_grad(p, Xr[:20], yr[:20])[0]
                p[k][i] = old
                num = (lp - lm) / 2e-6
                errs.append(abs(num - g[k][i]) / max(1e-8, abs(num) + abs(g[k][i])))
        print(f"gradient check: max relative error {max(errs):.1e}")

    v = [np.zeros_like(q) for q in p]          # SGD with momentum on the pretext task
    for _ in range(2000):
        i = rng.integers(0, len(yr), 64)
        g = loss_grad(p, Xr[i], yr[i])[1]
        for q, vq, gq in zip(p, v, g):
            vq *= 0.9; vq -= 0.05 * gq; q += vq
    pretext = ((features(p, Xr_test)[0] @ p[2] + p[3]).argmax(1) == yr_test).mean()

    # Downstream: name the shape (T, L, chevron) from 5 labelled upright images per class
    pool, kpool = make(600, rng)
    tr = np.concatenate([np.flatnonzero(kpool == c)[:5] for c in range(3)])
    Xtr, ytr = pool[tr], kpool[tr]
    Xte, yte = make(1000, rng)
    f = lambda q, X: features(q, X)[0]
    probes = {"raw pixels": linear_probe(Xtr.reshape(15, -1), ytr, Xte.reshape(1000, -1), yte),
              "untrained conv features": linear_probe(f(p_random, Xtr), ytr, f(p_random, Xte), yte),
              "rotation-pretrained": linear_probe(f(p, Xtr), ytr, f(p, Xte), yte)}
    return pretext, probes

if __name__ == "__main__":
    pretext, probes = run(0, check=True)
    print(f"rotation accuracy on held-out images: {pretext:.3f} (chance 0.25)")
    print("linear probe, 15 labelled images, 1000 test images:")
    for name, acc in probes.items():
        print(f"  {name:24s} {acc:.3f}")

gradient check: max relative error 4.5e-09
rotation accuracy on held-out images: 0.899 (chance 0.25)
linear probe, 15 labelled images, 1000 test images:
  raw pixels               0.756
  untrained conv features  0.630
  rotation-pretrained      0.939

Over seeds 0–9, rotation accuracy was 82–91%, and shape accuracy from 15 labels was 86–98% for the pretrained features, 68–81% for raw pixels and 38–75% for the untrained network; the pretrained features won on every seed. The untrained control shows the gain comes from pretext training, not the architecture alone. Our reading: recognizing an inverted T requires detectors for bars, stems and junctions in each orientation, and max pooling makes them position-independent, which raw pixels are not. With a fully connected hidden layer instead of the convolution, the average gain over raw pixels shrank to between −0.1 and 2.5 points in the three settings we tried.

Best Practices

• For two-view methods, compose at least random resized crops and color distortion, and check that the augmentations keep what the target task needs.
• Pre-train with a projection head and evaluate the layer before it.
• Monitor collapse: the per-dimension standard deviation of normalized embeddings over a batch, which SimSiam found stays near $1/\sqrt{d}$ in $d$ dimensions when outputs do not collapse, or k-NN accuracy on a small labelled set.
• Report several protocols with the exact backbone, epochs and resolution; numbers compare only under the same protocol.

Pitfalls

• Shortcuts: low-level cues (continuing edges, chromatic aberration, color histograms) solve the pretext task without useful features; high pretext accuracy proves little.
• Unsuitable invariances: rotation prediction assumes a canonical orientation, so in our reasoning it offers little signal for aerial or microscopy images.
• Silent collapse: the loss reaches its minimum while accuracy sits at chance, as in SimSiam without stop-gradient.

Evidence and Impact

He et al. (2020) concluded that the gap between unsupervised and supervised pre-training had been largely closed in many vision tasks. Reported ImageNet-1k top-1 accuracies:

• SimCLR: 69.3% linear with ResNet-50 and 76.5% with ResNet-50 (4×), which the authors described as matching a supervised ResNet-50 (Chen et al., 2020).
• MoCo: 60.6% linear with ResNet-50, but it beat supervised pre-training on seven detection and segmentation tasks on PASCAL VOC, COCO and other datasets (He et al., 2020). MoCo v2 reached 71.1% linear after 800 epochs with batch size 256 (Chen, Fan, Girshick & He, 2020).
• BYOL: 74.3% linear with ResNet-50 and 79.6% with a larger ResNet, without negatives (Grill et al., 2020).
• DINO: 80.1% linear with ViT-B/8 and 78.3% with k-NN for ViT-S/8 (Caron et al., 2021).
• MAE: 87.8% fine-tuned with ViT-H/14 at $448 \times 448$ (86.9% at 224), the best the authors found among methods using only ImageNet-1k data; it also beat supervised pre-training on COCO detection and segmentation and ADE20K semantic segmentation (He et al., 2022).
• DINOv2: 86.5% linear on frozen ViT-g/14 features at $224 \times 224$. In the same table, MAE’s ViT-H/14 reached 76.6% linear, which illustrates the gap between frozen and fine-tuned evaluation (Oquab et al., 2024, Table 4).

Architectures, epochs, augmentations and resolutions differ, so these numbers mark progress, not a controlled ranking.