SimCLR

SimCLR, “a simple framework for contrastive learning of visual representations”, is a self-supervised method that trains an image encoder without labels. Two random augmentations of each image in a batch form a positive pair, the other images’ views serve as negatives, and a ResNet followed by a small projection head is trained so that each view picks out its partner under the NT-Xent loss. Ting Chen, Simon Kornblith, Mohammad Norouzi and Geoffrey Hinton of Google Research’s Brain Team presented it at ICML 2020. Its paper is mostly a set of ablations: what makes the contrastive task useful, which augmentations, which head, which loss, which batch size. Their answers were strong composed augmentation, especially cropping with colour distortion; a nonlinear head whose input, not output, is kept; a normalized, temperature-scaled loss; and large batches with long training. With a ResNet-50 four times as wide, a linear classifier on the frozen features reached 76.5% top-1 accuracy on ImageNet, which the authors described as matching a supervised ResNet-50 (Chen et al., 2020). A follow-up, SimCLR v2 (Chen, Kornblith, Swersky, Norouzi & Hinton, 2020), scaled the method up for semi-supervised learning.

The Contrastive Learning article places SimCLR in the family and derives NT-Xent’s gradient and the role of the temperature; InfoNCE Loss covers the loss’s origin and its mutual-information bound. This article covers the method itself: its components and their ablations, the training recipe, the results, the colour-histogram shortcut that motivated colour distortion (with a worked example), the released code and SimCLR v2.

Problem Addressed

Chen et al. (2020, §1) divide unsupervised representation learning into generative methods, which model pixels and are expensive, and discriminative methods, which train on pretext tasks derived from the images. Hand-designed pretext tasks relied on heuristics, which, the authors argue, could limit the generality of the learned representations. Contrastive methods were giving the best results, but each needed something extra: a memory bank of stored features (instance discrimination, PIRL, MoCo) or a specialized architecture. As the paper describes them, AMDIM obtained global-to-local prediction by constraining the network’s receptive field, and CPC v1 and v2 predicted neighbouring patches with a fixed splitting of the image and a context aggregation network (§3; Appendix C).

SimCLR’s proposal was to move all of this into the data. A random crop with resizing produces, by chance, both kinds of task: a small crop inside a large one asks for global-to-local prediction, two neighbouring crops ask for adjacent-view prediction (their Figure 3). The predictive task is then decoupled from the network, which can be a standard ResNet, and the negatives come from the batch rather than from a memory bank. Under the linear evaluation protocol, the best ResNet-50 results the paper lists from earlier work were 63.8% for CPC v2 and 63.6% for PIRL, and 60.6% for MoCo (Table 6).

Architecture

SimCLR has four components (§2.1, Figure 2): a stochastic augmentation module, a base encoder $f$, a projection head $g$ and a contrastive loss. Each image $x$ gives two views $\tilde x_i = t(x)$ and $\tilde x_j = t'(x)$ with $t, t' \sim \mathcal{T}$ drawn independently from the same family. The encoder gives $h = f(\tilde x)$, the head gives $z = g(h)$, and the loss is applied to $z$. After training, $g$ is thrown away and $h$ is the representation.

Augmentation module

The default policy applies three operations in sequence (§2.1; Appendix A):

• Random crop and resize with a horizontal flip. The crop has a random area of 8–100% of the image and a random aspect ratio from 3/4 to 4/3, the standard Inception-style crop, and is resized to $224 \times 224$. A flip follows with probability 0.5. The authors call the flip helpful but not essential: removing it lowered the 100-epoch ResNet-50’s linear accuracy from 64.5% to 63.4%.
• Colour distortion with a strength $s$ ($s = 1$ by default). With probability 0.8, colour jitter changes brightness, contrast and saturation by up to $0.8s$ and hue by up to $0.2s$; then, with probability 0.2, the image is converted to grayscale (“colour drop”).
• Gaussian blur on half of the images, with $\sigma$ drawn from $[0.1, 2.0]$ and a kernel of 10% of the image side. It raised the same model from 63.2% to 64.5%.

The section on training below gives the ablations behind this choice. The appendix gives the colour distortion as TensorFlow pseudo-code; the released code implements it differently by default, a difference the example below prints.

Base encoder

The encoder is a standard ResNet, ResNet-50 by default, and $h$ is the 2,048-dimensional output of the final average pooling layer. The paper imposes no constraint on the architecture. In its Figure 7, deeper and wider ResNets all improved, and the gap between linear classifiers on SimCLR features and supervised models of the same size narrowed as models grew, which the authors read as unsupervised learning benefiting more from bigger models (§4.1). For the final comparisons they used ResNet-50 at width multipliers 1×, 2× and 4× (§6).

Projection head

The head is an MLP with one hidden layer, $z = W^{(2)} \sigma(W^{(1)} h)$ with a ReLU $\sigma$, mapping to a 128-dimensional $z$ (§2.1, §2.3). Figure 8 compared three heads at several output sizes: the nonlinear head beat a linear projection by about 3% and no projection by more than 10%, and with a head the output size made little difference. Even with the nonlinear head, the layer before it, $h$, was more than 10% better than $z$ as a representation (§4.2).

The authors conjectured that the loss makes $z$ invariant to the augmentations, so $g$ can discard information that downstream tasks need, such as colour or orientation, while $h$ keeps it. Their Table 3 trained MLPs to predict, from $h$ or from $g(h)$ (both 2,048-dimensional for this test), which transformation had been applied during pre-training:

What to predict	Random guess	From $h$	From $g(h)$
Colour vs grayscale	80	99.3	97.4
Rotation	25	67.6	25.6
Original vs corrupted	50	99.5	59.6
Original vs Sobel-filtered	50	96.6	56.3

Rotation, noise and Sobel filtering were added during pre-training only for the last three rows. Appendix B.4 adds that a linear head $W \in \mathbb{R}^{2048 \times 2048}$ has relatively few large eigenvalues, so it is approximately low-rank, and that in t-SNE plots of ten classes the classes are better separated in $h$ than in $z$.

Contrastive loss

For a batch of $N$ images, the $2N$ views give, for a positive pair $(i, j)$,

$$\ell_{i,j} = -\log \frac{\exp\big(\mathrm{sim}(z_i, z_j)/\tau\big)}{\sum_{k=1}^{2N} \mathbb{1}[k \ne i]\,\exp\big(\mathrm{sim}(z_i, z_k)/\tau\big)},$$

with $\mathrm{sim}$ the cosine similarity and $\tau$ a temperature (Eq. 1). Negatives are not sampled: the other $2(N-1)$ views of the batch serve as negatives. The loss is averaged over both orderings of every pair, $\mathcal{L} = \frac{1}{2N} \sum_{k=1}^{N} [\ell(2k-1, 2k) + \ell(2k, 2k-1)]$ (Algorithm 1). The authors note that the loss had been used before, citing Sohn (2016), Wu et al. (2018) and CPC, and named it NT-Xent, the normalized temperature-scaled cross-entropy. Its gradient, its hard-negative weighting and the temperature are derived in Contrastive Learning, and its reading as a mutual-information bound in InfoNCE Loss.

The paper compared NT-Xent with a logistic loss (NT-Logistic) and a margin triplet loss, all on $\ell_2$-normalized vectors and each with tuned hyperparameters. The gradients in its Table 2 show that only NT-Xent weights each negative by its relative hardness, so the other two were also run with semi-hard negative mining (§5.1). Linear accuracies in Table 4 were 50.9% (margin), 51.6% (NT-Logistic), 57.5% (margin, semi-hard), 57.9% (NT-Logistic, semi-hard) and 63.9% (NT-Xent); for simplicity, this comparison used negatives from one augmentation view only (Appendix B.10). Table 5 varied normalization and temperature with a contrastive distribution over 4,096 examples. Without $\ell_2$ normalization ($\tau$ = 10 or 100), the contrastive task was solved more often (91.7% and 92.1% contrastive accuracy) but linear accuracy fell to 57.2% and 57.0%, against 64.4% with normalization at $\tau = 0.1$.

Training

The default setting used in the ablations (§2.3) pre-trains a ResNet-50 on the ImageNet ILSVRC-2012 training set for 100 epochs at a batch size of 4,096, with the 2-layer head, NT-Xent and the LARS optimizer: learning rate $4.8 = 0.3 \times \text{BatchSize}/256$, weight decay $10^{-6}$, linear warm-up for 10 epochs and cosine decay without restarts. The authors note that 100 epochs do not reach the best performance but allow fair and efficient ablations. The final models trained for 1,000 epochs (§6).

• Large batches. The batch size ranged from 256 to 8,192; at 8,192 each positive pair has 16,382 negatives. LARS was used at every batch size, because standard SGD with momentum and linear learning-rate scaling may be unstable with large batches. Training ran on 32 to 128 Cloud TPU cores; a footnote gives about 1.5 hours for 100 epochs of ResNet-50 at batch 4,096 on 128 TPU v3 cores (§2.2).
• Global batch normalization. With data-parallel training, batch normalization usually computes its statistics per device. Because both views of a positive pair are on the same device, the authors argue that the model could use this local information leak to identify positives without learning better representations. SimCLR therefore aggregates the BN mean and variance over all devices; they name shuffling examples across devices (MoCo) and layer normalization (CPC v2) as alternatives (§2.2).

Batch size and training length

In Figure 9, a larger batch gave a large advantage at 100 epochs, and the gaps shrank or disappeared with more epochs, provided batches are randomly resampled (§5.2). The authors attribute this to negatives: a larger batch, or longer training, provides more of them. Appendix B.1 repeats the grid with a square-root learning-rate scaling, $0.075 \times \sqrt{\text{BatchSize}}$, which equals the linear rule at 4,096 (Table B.1, top-1):

Batch size	100 epochs, linear / sqrt	800 epochs, linear / sqrt
256	57.5 / 62.8	66.6 / 66.5
4,096	64.6 / 64.5	68.9 / 69.1
8,192	64.8 / 64.8	69.0 / 69.1

Square-root scaling helped small batches trained for few epochs. With batches up to 32,768 and up to 3,200 epochs, performance seemed to saturate at a batch of 8,192, while longer training still improved it (Figure B.2).

Composing augmentations

To study each augmentation without the confound of always-present cropping, the authors first cropped and resized every image and then applied the transformation under study to one branch only, leaving the other as the identity (§3.1). They note that this asymmetric setting hurts performance. In Figure 5, a single transformation never gave a good representation, although the model could almost perfectly identify the positives. The diagonal (one transformation) ran from 33.1% for crop to 2.6% for blur and rotation, colour alone gave 18.8%, and crop followed by colour gave 56.3% (55.8% in the other order), the best pair. Appendix B.2 reports that adding Sobel filtering, extra colour operations (equalize, solarize) and motion blur raised the final ResNet-50 (1×, 2×, 4×) linear accuracies by 0.7, 0.2 and 0.3 points, to 70.0%, 74.4% and 76.8%.

The authors’ conjecture for why crop and colour stand out is that most crops of one image share a similar colour distribution. Their Figure 6 plots histograms of pixel intensities, pooled over all channels, for several crops of two images: without colour distortion each image’s crops have nearly the same histogram, with it they do not. Colour histograms alone therefore suffice to tell images apart, and a network may exploit that shortcut (§3.1). Table 1 then varied the colour-distortion strength:

Colour distortion strength	1/8	1/4	1/2	1	1 (+ blur)	AutoAugment
SimCLR (linear)	59.6	61.0	62.6	63.2	64.5	61.1
Supervised	77.0	76.7	76.5	75.7	75.4	77.1

Stronger colour distortion improved the unsupervised model and did not help, or hurt, the supervised one trained with the same augmentations; AutoAugment, a policy searched with supervised learning, did no better than crop with strong colour distortion. The authors conclude that contrastive learning benefits from stronger augmentation than supervised learning (§3.2). Appendix B.3 checks the supervised side with longer training: a supervised ResNet-50 reached 75.3% after 90 epochs and 76.7% after 500 with crop, colour and blur, against 76.5% and 76.2% with crop alone.

Example: the colour-histogram shortcut

The example measures the shortcut of §3.1 without training anything. The encoders are fixed histograms, so whatever they achieve on the contrastive task is available to any network that computes such statistics.

• Instances. Every non-overlapping $128 \times 128$ tile of six RGB photographs bundled with scikit-image (astronaut, chelsea, coffee, rocket, immunohistochemistry and the left image of the motorcycle stereo pair) is one instance: 80 instances, one batch of $2N = 160$ views. In our reading, tiles from one photograph are each other’s hardest negatives.
• Views. Two views per tile, made as in Appendix A and the released code: an Inception-style crop resized to $64 \times 64$ with bicubic interpolation, a flip, and, depending on the condition, the colour distortion. The crops, flips and random colour parameters of each view are drawn once and reused in every condition, so conditions differ only in the colour operations. Twenty seeds (0–19) give twenty batches.
• Conditions. Crop only; crop plus colour distortion at the four strengths of Table 1; and, at $s = 1$, the released code’s default colour jitter. The appendix pseudo-code adds a random offset to brightness, applies the four jitter operations in a fixed order (a comment notes that the order can also be shuffled) and clips once at the end. The released code’s default multiplies brightness by a random factor (an option the code labels “simclrv2”, against “simclrv1” for the additive form), shuffles the order and clips after every operation (data_util.py at the pinned commit). The sources thus differ on this modelling choice, so both are printed. The jitter operations follow the TensorFlow docstrings: contrast scales each channel about its mean, saturation scales S in HSV and clips it, hue rotates H, and grayscale uses TensorFlow’s weights $(0.2989, 0.5870, 0.1140)$.
• Encoders. (1) A 32-bin histogram of pixel intensities pooled over the three channels, Figure 6’s quantity; (2) a joint RGB histogram with $8 \times 8 \times 8$ bins; (3) a colour-blind control, the histogram of the ten uniform local binary pattern codes ($P = 8$, $R = 1$) of the grey view. The bin counts are our choice.
• Measures. Top-1 retrieval, the fraction of the 160 views whose most cosine-similar other view is their partner (ties count as misses), and the exact NT-Xent of Eq. 1 at $\tau = 0.1$, the best value in Table 5 and the code’s default, and $\tau = 0.5$, the best for CIFAR-10 in Appendix B.9. Chance is $1/159 = 0.0063$ for retrieval, and an encoder that cannot tell views apart pays $\ln 159 = 5.0689$.
• Checks. The vectorized loss against Eq. 1 and Algorithm 1 written term by term; a constant embedding, which must give exactly $\ln(2N - 1)$ and no correct retrieval; the colour-space conversion against scikit-image and two identities of the jitter; identical views, which every encoder must match perfectly (so each tile’s histogram is unique and the partners are identifiable); and the crop statistics.

The planned hypotheses were: (H1) with crop only, both colour histograms retrieve at least half the partners and their NT-Xent at $\tau = 0.1$ is at least 1 nat below chance; (H2) for both, retrieval falls and NT-Xent rises at every step of strength, retrieval at $s = 1$ is below half its crop-only value, and the best seed at $s = 1$ is below the worst seed with crop only; (H3) the colour-blind LBP encoder loses a smaller share of its retrieval from crop only to $s = 1$ than the colour histograms; (H4) with crop only the joint RGB histogram beats the pooled one; (H5) the code-default jitter also leaves both colour histograms below half their crop-only retrieval. The script uses NumPy 2.3.5, SciPy 1.16.3 and scikit-image 0.25.2 on one BLAS thread and takes about 30 seconds; two runs printed identical output.

import os
for v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS", "VECLIB_MAXIMUM_THREADS"):
    os.environ[v] = "1"                                   # one BLAS thread, set before NumPy loads
import numpy as np
from scipy.special import logsumexp
from skimage import data, color
from skimage.transform import resize
from skimage.feature import local_binary_pattern

T, OUT, SEEDS = 128, 64, range(20)                        # tile side, view side, augmentation seeds
STRENGTHS = [0.125, 0.25, 0.5, 1.0]                       # colour-distortion strengths s (Table 1)
GRAY = np.array([0.2989, 0.5870, 0.1140])                 # tf.image.rgb_to_grayscale weights

# ---- data: every non-overlapping 128 x 128 tile of six bundled RGB photographs is one instance
photos = {"astronaut": data.astronaut(), "chelsea": data.chelsea(), "coffee": data.coffee(),
          "rocket": data.rocket(), "ihc": data.immunohistochemistry(),
          "motorcycle": data.stereo_motorcycle()[0]}
tiles, source = [], []
for name, im in photos.items():
    im = im.astype(float) / 255
    for r in range(im.shape[0] // T):
        for c in range(im.shape[1] // T):
            tiles.append(im[r * T:(r + 1) * T, c * T:(c + 1) * T])
            source.append(name)
N = len(tiles)

# ---- RGB <-> HSV on arbitrary floats (the conversion the TF colour ops describe)
def rgb_to_hsv(x):
    v = x.max(-1)
    rng = v - x.min(-1)
    s = np.where(v > 0, rng / np.where(v > 0, v, 1), 0.0)
    safe = np.where(rng > 0, rng, 1)
    r, g, b = x[..., 0], x[..., 1], x[..., 2]
    h = np.where(v == r, (g - b) / safe, np.where(v == g, 2 + (b - r) / safe, 4 + (r - g) / safe))
    h = np.where(rng > 0, (h / 6) % 1.0, 0.0)
    return np.stack([h, s, v], -1)

def hsv_to_rgb(hsv):
    h, s, v = hsv[..., 0] % 1.0, hsv[..., 1], hsv[..., 2]
    h6 = h * 6
    c = v * s
    xx = c * (1 - np.abs(h6 % 2 - 1))
    m = v - c
    z = np.zeros_like(h)
    k = np.floor(h6).astype(int) % 6
    rgb = np.select([k[..., None] == i for i in range(6)],
                    [np.stack(t, -1) for t in ((c, xx, z), (xx, c, z), (z, c, xx),
                                               (z, xx, c), (xx, z, c), (c, z, xx))])
    return rgb + m[..., None]

# ---- SimCLR augmentations
def crop_box(rng, side):
    """Inception-style crop (Appendix A): area 8-100% of the tile, aspect ratio 3/4-4/3."""
    for _ in range(100):
        area = rng.uniform(0.08, 1.0) * side * side
        ar = rng.uniform(3 / 4, 4 / 3)
        w, h = int(round(np.sqrt(area * ar))), int(round(np.sqrt(area / ar)))
        if 0 < w <= side and 0 < h <= side:
            return rng.integers(0, side - h + 1), rng.integers(0, side - w + 1), h, w
    return 0, 0, side, side                                # fallback: the whole tile

def crop_resize_flip(tile, box, flip):
    y, x, h, w = box
    v = resize(tile[y:y + h, x:x + w], (OUT, OUT), order=3, anti_aliasing=False, mode="edge")
    v = np.clip(v, 0, 1)
    return v[:, ::-1] if flip else v

def colour_jitter_v1(x, s, u):
    """Appendix A's TensorFlow pseudo-code: additive brightness, contrast, saturation, hue in
    this order, one clip at the end. u holds four uniforms in [-1, 1) for this view."""
    x = x + 0.8 * s * u[0]                                     # random_brightness, max_delta 0.8 s
    mean = x.mean((0, 1), keepdims=True)
    x = (x - mean) * (1 + 0.8 * s * u[1]) + mean               # random_contrast, factor in [1 -/+ 0.8 s)
    hsv = rgb_to_hsv(x)
    hsv[..., 1] = np.clip(hsv[..., 1] * (1 + 0.8 * s * u[2]), 0, 1)   # random_saturation
    x = hsv_to_rgb(hsv)
    hsv = rgb_to_hsv(x)
    hsv[..., 0] = hsv[..., 0] + 0.2 * s * u[3]                 # random_hue, delta in [-0.2 s, 0.2 s)
    return np.clip(hsv_to_rgb(hsv), 0, 1)

def colour_jitter_code(x, s, u, order):
    """The released code's default: multiplicative brightness ('simclrv2'), random order,
    a clip after every operation."""
    for op in order:
        if op == 0:
            x = x * (1 + 0.8 * s * u[0])
        elif op == 1:
            mean = x.mean((0, 1), keepdims=True)
            x = (x - mean) * (1 + 0.8 * s * u[1]) + mean
        elif op == 2:
            hsv = rgb_to_hsv(x)
            hsv[..., 1] = np.clip(hsv[..., 1] * (1 + 0.8 * s * u[2]), 0, 1)
            x = hsv_to_rgb(hsv)
        else:
            hsv = rgb_to_hsv(x)
            hsv[..., 0] = hsv[..., 0] + 0.2 * s * u[3]
            x = hsv_to_rgb(hsv)
        x = np.clip(x, 0, 1)
    return x

def to_gray3(x):
    return np.repeat((x @ GRAY)[..., None], 3, -1)

def colour_distort(x, s, draw, impl):
    """random_apply(color_jitter, p=0.8) then random_apply(to_grayscale, p=0.2)."""
    if draw["p_jit"] < 0.8:
        x = colour_jitter_v1(x, s, draw["u"]) if impl == "v1" else \
            colour_jitter_code(x, s, draw["u"], draw["order"])
    if draw["p_gray"] < 0.2:
        x = to_gray3(x)
    return x

# ---- fixed encoders (nothing is trained)
def enc_intensity(x):                       # Figure 6: pixel intensities pooled over the channels
    return np.histogram(x, bins=32, range=(0, 1))[0].astype(float)

def enc_rgb(x):                             # joint colour histogram, 8 bins per channel
    q = np.minimum((x * 8).astype(int), 7)
    return np.bincount((q[..., 0] * 64 + q[..., 1] * 8 + q[..., 2]).ravel(), minlength=512).astype(float)

def enc_lbp(x):                             # colour-blind: uniform LBP codes of the grey image
    g = np.round(255 * np.clip(x @ GRAY, 0, 1)).astype(np.uint8)
    codes = local_binary_pattern(g, P=8, R=1, method="uniform")
    return np.bincount(codes.astype(int).ravel(), minlength=10).astype(float)

ENCODERS = {"intensity hist": enc_intensity, "RGB hist": enc_rgb, "grey LBP hist": enc_lbp}

# ---- NT-Xent (Eq. 1, averaged over all 2N views as in Algorithm 1) and positive retrieval
def nt_xent(Z, tau):
    U = Z / np.linalg.norm(Z, axis=1, keepdims=True)
    L = U @ U.T / tau
    np.fill_diagonal(L, -np.inf)                           # the indicator k != i
    p = np.arange(len(Z)) ^ 1                              # rows 2k and 2k+1 are a positive pair
    return float(np.mean(logsumexp(L, axis=1) - L[np.arange(len(Z)), p]))

def retrieval(Z):
    """Fraction of views whose most similar other view is their partner (ties count as misses)."""
    U = Z / np.linalg.norm(Z, axis=1, keepdims=True)
    S = U @ U.T
    np.fill_diagonal(S, -np.inf)
    p = np.arange(len(Z)) ^ 1
    pos = S[np.arange(len(Z)), p].copy()
    S[np.arange(len(Z)), p] = -np.inf
    return float(np.mean(pos > S.max(1)))

def nt_xent_loop(Z, tau):
    """Eq. 1 and Algorithm 1 written out term by term, as a check on nt_xent."""
    U = Z / np.linalg.norm(Z, axis=1, keepdims=True)
    n2 = len(Z)
    def ell(i, j):
        den = sum(np.exp(U[i] @ U[k] / tau) for k in range(n2) if k != i)
        return -np.log(np.exp(U[i] @ U[j] / tau) / den)
    return sum(ell(2 * k, 2 * k + 1) + ell(2 * k + 1, 2 * k) for k in range(n2 // 2)) / n2

# ---- checks
print(f"data: {N} tiles of {T}x{T} from " + ", ".join(f"{k} {source.count(k)}" for k in photos))
print(f"  pixel range [{min(t.min() for t in tiles):.3f}, {max(t.max() for t in tiles):.3f}]; "
      f"chance: retrieval 1/(2N-1) = {1 / (2 * N - 1):.4f}, NT-Xent ln(2N-1) = {np.log(2 * N - 1):.4f}")
rng = np.random.default_rng(123)
Zr = rng.random((2 * 12, 7))
print(f"check NT-Xent vectorized vs term by term: {abs(nt_xent(Zr, 0.1) - nt_xent_loop(Zr, 0.1)):.1e}")
print(f"check constant embedding: NT-Xent - ln(2N-1) = {nt_xent(np.ones((2 * N, 5)), 0.1) - np.log(2 * N - 1):.1e}, "
      f"retrieval {retrieval(np.ones((2 * N, 5))):.4f}")
x = tiles[0]
same = rgb_to_hsv(x)
print(f"check HSV round trip {np.abs(hsv_to_rgb(same) - x).max():.1e}; vs skimage rgb2hsv "
      f"{np.abs(same - color.rgb2hsv(x)).max():.1e}; s = 0 jitter is identity "
      f"{np.abs(colour_jitter_v1(x, 0.0, np.ones(4)) - x).max():.1e}; hue +1 is identity "
      f"{np.abs(hsv_to_rgb(same + [1.0, 0, 0]) - x).max():.1e}")
for name, f in ENCODERS.items():
    Z = np.stack([f(t) for t in tiles for _ in range(2)])      # identical views: no augmentation
    print(f"check identical views, {name}: retrieval {retrieval(Z):.4f}")

# ---- the experiment
conds = ["crop only"] + [f"crop + colour s={s:g}" for s in STRENGTHS] + ["crop + colour s=1, code default"]
res = {(c, e): [] for c in conds for e in ENCODERS}
areas, aspects, fallbacks = [], [], 0
for seed in SEEDS:
    rng = np.random.default_rng(seed)
    views = []
    for t in tiles:
        for _ in range(2):
            box = crop_box(rng, T)
            fallbacks += box[2] == T and box[3] == T and box[0] == 0 and box[1] == 0
            areas.append(box[2] * box[3] / T**2)
            aspects.append(box[3] / box[2])
            v = crop_resize_flip(t, box, rng.random() < 0.5)
            draw = {"p_jit": rng.random(), "p_gray": rng.random(),
                    "u": rng.uniform(-1, 1, 4), "order": rng.permutation(4)}
            views.append((v, draw))
    batches = {"crop only": [v for v, _ in views]}
    for s in STRENGTHS:
        batches[f"crop + colour s={s:g}"] = [colour_distort(v, s, d, "v1") for v, d in views]
    batches["crop + colour s=1, code default"] = [colour_distort(v, 1.0, d, "code") for v, d in views]
    for c in conds:
        for e, f in ENCODERS.items():
            Z = np.stack([f(v) for v in batches[c]])
            res[c, e].append((retrieval(Z), nt_xent(Z, 0.1), nt_xent(Z, 0.5)))

areas, aspects = np.array(areas), np.array(aspects)
print(f"crops: area fraction [{areas.min():.3f}, {areas.max():.3f}], aspect [{aspects.min():.3f}, "
      f"{aspects.max():.3f}], fallbacks {fallbacks}")
print(f"\n{len(SEEDS)} seeds; 2N = {2 * N} views per batch; mean [min, max] over seeds")
print(f"{'condition':33s}{'encoder':16s}{'top-1 retrieval':>26s}{'NT-Xent tau=0.1':>26s}{'NT-Xent tau=0.5':>26s}")
for c in conds:
    for e in ENCODERS:
        a = np.array(res[c, e])
        print(f"{c:33s}{e:16s}" + "".join(f"{m:10.4f} [{lo:.4f}, {hi:.4f}]"
              for m, lo, hi in zip(a.mean(0), a.min(0), a.max(0))))
print("relative fall in mean retrieval, crop only -> crop + colour s=1: " + ", ".join(
    f"{e} {1 - np.mean(res['crop + colour s=1', e], 0)[0] / np.mean(res['crop only', e], 0)[0]:.4f}"
    for e in ENCODERS))

data: 80 tiles of 128x128 from astronaut 16, chelsea 6, coffee 12, rocket 15, ihc 16, motorcycle 15
  pixel range [0.000, 1.000]; chance: retrieval 1/(2N-1) = 0.0063, NT-Xent ln(2N-1) = 5.0689
check NT-Xent vectorized vs term by term: 8.9e-16
check constant embedding: NT-Xent - ln(2N-1) = 0.0e+00, retrieval 0.0000
check HSV round trip 2.1e-16; vs skimage rgb2hsv 0.0e+00; s = 0 jitter is identity 2.5e-16; hue +1 is identity 4.2e-16
check identical views, intensity hist: retrieval 1.0000
check identical views, RGB hist: retrieval 1.0000
check identical views, grey LBP hist: retrieval 1.0000
crops: area fraction [0.080, 0.992], aspect [0.742, 1.343], fallbacks 0

20 seeds; 2N = 160 views per batch; mean [min, max] over seeds
condition                        encoder                    top-1 retrieval           NT-Xent tau=0.1           NT-Xent tau=0.5
crop only                        intensity hist      0.6228 [0.5437, 0.6937]    2.6046 [2.4398, 2.8178]    4.2162 [4.1748, 4.2671]
crop only                        RGB hist            0.7113 [0.6750, 0.7812]    1.5797 [1.3315, 1.8079]    3.6836 [3.6309, 3.7412]
crop only                        grey LBP hist       0.2206 [0.1625, 0.3250]    4.4437 [4.3548, 4.5608]    4.8604 [4.8397, 4.8832]
crop + colour s=0.125            intensity hist      0.1416 [0.0813, 0.2125]    4.7745 [4.2688, 5.3021]    4.6433 [4.5246, 4.7534]
crop + colour s=0.125            RGB hist            0.2153 [0.1437, 0.3000]    4.9616 [3.5978, 5.6009]    4.4498 [4.1913, 4.5810]
crop + colour s=0.125            grey LBP hist       0.1803 [0.1250, 0.3125]    4.4608 [4.3722, 4.5929]    4.8636 [4.8389, 4.8932]
crop + colour s=0.25             intensity hist      0.0834 [0.0187, 0.1750]    5.9080 [5.1876, 6.4410]    4.8435 [4.6906, 4.9460]
crop + colour s=0.25             RGB hist            0.1187 [0.0500, 0.2000]    6.1087 [4.9160, 6.8227]    4.7155 [4.4965, 4.8669]
crop + colour s=0.25             grey LBP hist       0.1391 [0.0750, 0.2062]    4.5462 [4.4103, 4.7221]    4.8731 [4.8534, 4.8973]
crop + colour s=0.5              intensity hist      0.0619 [0.0187, 0.1250]    7.2155 [6.4420, 7.7242]    5.0361 [4.8831, 5.1528]
crop + colour s=0.5              RGB hist            0.0650 [0.0125, 0.1250]    6.9995 [5.8087, 7.9641]    4.9173 [4.7000, 5.1017]
crop + colour s=0.5              grey LBP hist       0.0919 [0.0437, 0.1500]    4.8544 [4.6366, 5.2223]    4.9081 [4.8722, 4.9632]
crop + colour s=1                intensity hist      0.0378 [0.0125, 0.0813]    8.5291 [7.7974, 9.0947]    5.2059 [5.0588, 5.3355]
crop + colour s=1                RGB hist            0.0416 [0.0125, 0.0813]    8.0770 [7.1934, 8.9374]    5.1212 [4.9560, 5.2932]
crop + colour s=1                grey LBP hist       0.0488 [0.0125, 0.1000]    5.7015 [5.2537, 6.1794]    5.0218 [4.9307, 5.0899]
crop + colour s=1, code default  intensity hist      0.0513 [0.0125, 0.1000]    7.0438 [6.5259, 7.7351]    4.9838 [4.8681, 5.1363]
crop + colour s=1, code default  RGB hist            0.0559 [0.0125, 0.1125]    7.0940 [6.2587, 7.7759]    4.9453 [4.8008, 5.1035]
crop + colour s=1, code default  grey LBP hist       0.0725 [0.0125, 0.1500]    4.9306 [4.7563, 5.1325]    4.9339 [4.8972, 4.9719]
relative fall in mean retrieval, crop only -> crop + colour s=1: intensity hist 0.9393, RGB hist 0.9416, grey LBP hist 0.7790

Every check passes: the two forms of the loss agree to $9 \times 10^{-16}$, the constant embedding gives exactly the chance loss, the colour conversions are exact to $4 \times 10^{-16}$, identical views are always matched, and the crops stay within the appendix’s ranges, up to the rounding of their sides to whole pixels.

The shortcut exists (H1 held). With crop only, the joint RGB histogram finds the partner for 71% of views on average and the pooled intensity histogram for 62%, against 0.6% by chance; even their worst seeds stay above half. Their NT-Xent at $\tau = 0.1$ is 3.5 and 2.5 nats below the chance value. Neither encoder has seen any training: crops of one tile carry the tile’s colour statistics, and that alone solves most of the contrastive task. The figure is below the near-perfect identification the paper reports for networks trained on single transformations, which in our reading reflects the hard negatives here, tiles cut from the same photograph.

Colour distortion removes it (H2 held). For both colour histograms, mean retrieval falls and both losses rise at every step from crop only through $s = 1/8, 1/4, 1/2$ to $1$; at $s = 1$ retrieval is about 4%, roughly 6% of its crop-only value, and the best seed at $s = 1$ (0.0813) is far below the worst crop-only seed (0.5437 and 0.6750). The steps hold for the means; beyond the first step, the retrieval ranges over seeds of neighbouring strengths overlap, and so do the loss ranges, except the pooled histogram’s at $\tau = 0.1$ from $s = 1/4$ to $1/2$ and from $1/2$ to $1$. The steepest fall is the first: even $s = 1/8$ removes more than two-thirds of the retrieval, while in Table 1 the linear accuracy kept rising up to $s = 1$. The histogram shortcut is a single cue that a mild distortion already breaks; what the network learns instead once it is broken is not something this example can show.

The control loses less (H3 held). The grey LBP histogram loses 78% of its retrieval from crop only to $s = 1$, against 94% for both colour histograms. It starts much lower, 22% with crop only, in our reading because the crops change the scale of the texture it counts, and from $s = 1/4$ on it retrieves better than either colour histogram. It does not escape the distortion: at $s = 1$ it is at 5%. In our reading, hue and saturation changes alter the grey values unevenly across pixels and clipping flattens bright or dark regions, so the LBP codes change too; the colour distortion is not neutral to structure, only less damaging to it.

Joint against pooled (H4 held in the mean). With crop only, the joint RGB histogram beats the pooled intensity histogram by 9 points on average, 0.7113 against 0.6228, but the seed ranges overlap (0.6750–0.7812 against 0.5437–0.6937). In this example, Figure 6’s pooled histogram is the weaker of the two colour cues.

Two implementations of the jitter (H5 held). The code-default jitter at $s = 1$ also leaves both colour histograms far below half their crop-only retrieval (0.0513 and 0.0559). It is milder than the appendix’s version for all three encoders, with higher mean retrieval (0.0513 against 0.0378, 0.0559 against 0.0416, 0.0725 against 0.0488) and lower loss at $\tau = 0.1$. The retrieval ranges over seeds overlap for all three; the loss ranges at $\tau = 0.1$ are disjoint for the pooled and LBP histograms and overlap for the joint RGB histogram. In our reading, a multiplicative brightness keeps dark pixels dark and leaves fewer pixels at the clipping limits than an additive shift of up to $\pm 0.8$.

A loss above chance. At $\tau = 0.1$, both colour histograms have a mean NT-Xent above $\ln 159 = 5.0689$ from $s = 1/4$ on, up to 8.53 for the pooled histogram at $s = 1$; the LBP histogram exceeds it at $s = 1$ (5.7015). At $\tau = 0.5$ only the two colour histograms at $s = 1$ exceed it (5.2059 and 5.1212). The following derivation is ours. The log of a sum of $2N - 1$ exponentials is at least $\ln(2N - 1)$ plus their mean exponent (Jensen’s inequality), so each view’s term satisfies

$$\ell_i \ge \ln(2N - 1) + \frac{\bar s_i - s_{i,p(i)}}{\tau},$$

where $s_{i,p(i)}$ is the similarity to the partner and $\bar s_i$ the mean similarity to all $2N - 1$ other views. A view whose partner is less similar than the average view pays more than chance, by at least the gap divided by $\tau$. In our reading, strong distortion makes many partners less similar than average, for instance when only one view of a pair is converted to grey; the factor $1/\tau$ explains why the excess appears at $\tau = 0.1$ before $\tau = 0.5$.

The example shows that the cue is there, not that SimCLR’s network would use it; the paper says only that networks may exploit it. It also leaves out Gaussian blur, part of the default policy.

Inference

At inference the projection head is discarded and $h$ is used. The paper evaluates $h$ in three ways (§2.3, §6; Appendix B.5–B.8), the protocols of Evaluating Representations:

• Linear evaluation. A linear classifier on the frozen encoder, trained with Nesterov momentum at batch 4,096 and learning rate 1.6 for 90 epochs; LARS with the pre-training hyperparameters gave similar results, and so did training the linear classifier jointly during pre-training with a stop-gradient on its input (Appendix B.6).
• Semi-supervised fine-tuning. The whole network is fine-tuned on 1% or 10% of the ImageNet labels, sampled per class (about 12.8 and 128 images per class), with Nesterov momentum at batch 4,096, learning rate 0.8, no warm-up and no regularization, for 60 or 30 epochs; only random crops and flips are used, and test images are resized to 256 and centre-cropped to 224 (Appendix B.5).
• Transfer. On 12 classification datasets, either an $\ell_2$-regularized multinomial logistic regression on frozen features (fitted with L-BFGS) or full fine-tuning, with hyperparameters selected per dataset on a validation set, following Kornblith et al. (Appendix B.8).

Performance

All results are ImageNet ILSVRC-2012 validation top-1 accuracy unless stated, after 1,000 epochs of pre-training (Chen et al., 2020).

• Linear evaluation (Table 6). ResNet-50: 69.3% top-1 (89.0% top-5); ResNet-50 (2×): 74.2%; ResNet-50 (4×): 76.5% (93.2% top-5). The best earlier entry in the table was CPC v2 with a modified ResNet-161 at 71.5%. Appendix B.8 gives the supervised baselines trained with the same augmentations for 1,000 epochs: 76.3% for ResNet-50 and 78.3% for ResNet-50 (4×), gaps of 7.0 and 1.8 points by our arithmetic (the appendix’s text says 6.8 for the first).
• Few labels (Table 7, top-5; Table B.4, top-1). Fine-tuned on 1% of the labels, ResNet-50 (4×) reached 85.8% top-5 (63.0% top-1), and on 10%, 92.6% top-5 (74.4% top-1); the standard ResNet-50 reached 75.5% and 87.8% top-5 (48.3% and 65.6% top-1). The supervised baseline the paper quotes reached 48.4% and 80.4% top-5. CPC v2 with its ResNet-161 had the higher top-1 with 1% of labels, 52.7% against the standard ResNet-50’s 48.3%.
• Full fine-tuning (Table B.2). Pre-trained with the broader augmentations and fine-tuned on all labels, ResNet-50 (4×) reached 80.4% top-1 against 78.4% when trained from scratch; for the standard ResNet-50, fine-tuning gave no improvement.
• Transfer (Table 8, ResNet-50 (4×)). Fine-tuned, the SimCLR model was significantly better than the supervised baseline on 5 of 12 datasets and worse on 2 (Pets and Flowers), and tied on the rest, under a permutation test. With the standard ResNet-50 (Table B.5), the supervised model had the higher accuracy on all 12 datasets under linear evaluation (on DTD the table marks the two as not significantly different) and on 9 of 12 with fine-tuning, of which the table marks 6 as significant; the appendix’s text says the supervised model is better on all datasets with linear evaluation and on 10 of 12 with fine-tuning (Appendix B.8.2).
• CIFAR-10 (Appendix B.9). Pre-trained on CIFAR-10 itself, with a ResNet-50 whose stem was adapted to small images (a $3 \times 3$ stride-1 first convolution, no max pooling) and colour strength 0.5, the best model reached 94.0% linear accuracy against 95.1% for the supervised baseline; at convergence, $\tau = 0.5$ was the best of $\{0.1, 0.5, 1.0\}$, consistently, the authors note, across batch sizes.

Later papers re-ran SimCLR under shared protocols. In Chen and He’s SimSiam paper (2021, Table 4), their improved reproduction of SimCLR (ResNet-50, batch 4,096) reached 66.5%, 68.3%, 69.8% and 70.4% after 100, 200, 400 and 800 epochs, below their reproductions of BYOL (74.3% at 800 epochs) and of MoCo v2 at batch 256 (72.2%), and below SimSiam at every length. Chen, Xie and He (2021, Table 4) trained Vision Transformers with SimCLR: 69.0% for ViT-S/16 and 73.9% for ViT-B/16 after 300 epochs, against 72.5% and 76.5% for MoCo v3.

Strengths

• Simplicity. A standard ResNet, an MLP head and an in-batch loss; no memory bank, momentum encoder or architecture tied to the pretext task. The authors attribute the results to the composition of known components rather than to any single one (§7).
• Explicit ablations. The paper isolates the effects of augmentation, head, loss, temperature, batch size and model size, which made the recipe easy to adopt and modify.
• Transfer at scale. With a wide encoder, linear and fine-tuned transfer reached the level of supervised pre-training on several datasets (Table 8).

Limitations

• Large batches and hardware. The recipe was tuned at batch 4,096 on TPUs, with 16,382 negatives per pair at 8,192. Small batches did worse at short schedules (Table B.1); the official README notes that a batch of 4,096 needs at least 32 TPUs, and that results tuned at large batch are suboptimal at small batch unless the learning rate, temperature and head depth are retuned. SimSiam’s authors note that SimCLR requires a large batch to work well (Chen & He, 2021, §2).
• Dependence on augmentation. The representation is only as good as the augmentations that define the task: no single transformation sufficed, and cues shared by crops, such as colour statistics, must be destroyed explicitly. Which invariances are right depends on the downstream task; $z$ discards colour and orientation information that $h$ keeps (Table 3).
• Negatives. Every other image counts as a negative, including other images of the same kind of object; Contrastive Learning discusses these false negatives and the temperature’s role.
• Narrow encoders lag supervised training. With the standard ResNet-50, supervised pre-training stayed ahead in linear evaluation (76.3% against 69.3%) and in most transfer tasks (Appendix B.8.2).

Historical Importance

SimCLR appeared in 2020, the same year as MoCo and its revision MoCo v2, and the two lines exchanged ideas: as the Contrastive Learning article describes, MoCo v2 adopted SimCLR’s MLP head and blur. Its ablations made three points: augmentation defines the task, a nonlinear projection head belongs between the representation and the loss, and the representation is taken before that head. Later methods reused the head: MoCo v2, as above, and MoCo v3, whose MLP heads put batch normalization in every layer “following SimCLR” (Chen, Xie & He, 2021). Chen and He (2021) placed SimCLR in a family of Siamese architectures (SimCLR, BYOL, SwAV, SimSiam) that share an encoder across two views and differ in how they avoid collapse; they describe SimSiam as “SimCLR without negatives”. Adding SimSiam’s predictor and stop-gradient to their SimCLR reproduction did not help (66.5%, 66.4% with the predictor, 66.0% with both).

SimCLR also became a standard pre-training method in applied work. In medicine, Azizi et al. (2021) pre-trained ResNets with SimCLR on ImageNet and then on unlabelled medical images (Medical Image Classification), and DSMIL trains its tile encoder with SimCLR on the training slides (Whole-Slide Image Classification). CLIP, by contrast, kept a contrastive loss but used no nonlinear projection head.

Variants

SimCLR v2. Chen, Kornblith, Swersky, Norouzi and Hinton (2020) presented SimCLR v2 at NeurIPS 2020. That paper could not be read for this article; what follows comes from readable sources, each named.

• The pipeline. Murphy (2022, §19.3.7) describes the semi-supervised recipe: self-supervised pre-training with SimCLR on the unlabelled data, fine-tuning on a small labelled set, and then distillation of the fine-tuned teacher’s predictions on the unlabelled data into a student, with a cross-entropy whose softmax has a temperature. He adds that the student is normally smaller than the teacher; when it has the same form, the procedure is self-training. The Semi-Supervised Learning article summarizes the same pipeline, and Knowledge Distillation covers the distillation step.
• Models. The official README lists 65 released models, corresponding, in its words, to Table 1 of the SimCLR v2 paper: ResNets of depth 50, 101 and 152, widths 1× to 3×, with and without selective kernels (SK). Its largest, a 152-layer, 3×-wide SK ResNet with 795 million parameters, is listed with 79.8 in the linear-evaluation column and 74.9, 80.1 and 83.1 after fine-tuning on 1%, 10% and 100% of the labels. Chen, Xie and He (2021) quote the same 79.8% as the best earlier ImageNet linear-probing result, from SimCLR v2’s Table 1, and describe selective kernels as a form of attention. The DINO paper’s comparison lists this model with 794 million parameters, 79.8% linear and 73.1% k-NN accuracy (DINO paper).
• Head and encoder. The README and run.py describe a projection head with a configurable number of layers (3 by default at the pinned commit) and fine-tuning from a chosen layer of it (run.py’s default, 0, discards the head; the README’s 1%-label command fine-tunes from its first layer); the README tells users to set these “following SimCLRv2 paper”. Chen, Xie and He (2021) note that SimCLR v2, like BYOL, uses a momentum encoder.
• Other changes. The README notes that the v1 and v2 models were pre-trained with different weight decays, so the convolutional weights of the v1 ResNet-50 are on average 16.8 times larger, and fine-tuning them with momentum SGD needs different hyperparameters. The released code’s default brightness jitter is the multiplicative form it labels “simclrv2”.

Related methods. MoCo keeps negatives in a queue encoded by a momentum encoder, BYOL and SimSiam drop negatives altogether, and supervised contrastive learning treats same-class images as positives; Contrastive Learning and Self-Supervised Learning compare them.

Implementations

The official implementation is google-research/simclr, in TensorFlow (a TF2 version sits in its tf2/ folder), under the Apache 2.0 licence; the README links released v1 and v2 checkpoints and third-party repositories that convert them to PyTorch. Reading the code at commit 383d414 (as of November 2026, the latest commit, from May 2023) shows several differences from the paper:

• The loss is doubled. objective.py computes two cross-entropies with TensorFlow’s softmax_cross_entropy, whose default reduction divides the sum by the number of non-zero weights (here the $N$ rows of one augmentation), and returns their sum, which is twice Algorithm 1’s $\mathcal{L}$. In our reading this only rescales the gradient, which a matching change of learning rate and weight decay absorbs.
• Defaults are v2-era. At this commit, run.py defaults to a 3-layer projection head, weight decay $10^{-4}$, temperature 0.1, colour strength 1.0 and global BN; the paper’s default setting used a 2-layer head and weight decay $10^{-6}$. The README’s ImageNet command uses learning rate 0.075 with square-root scaling, weight decay $10^{-4}$ and temperature 0.1; its CIFAR-10 command uses temperature 0.5, colour strength 0.5 and no blur.
• Colour jitter. data_util.py defaults to multiplicative brightness and a random order of the four jitter operations, with a clip after each; the appendix pseudo-code uses additive brightness, a fixed order and one final clip. The pseudo-code’s colour-drop function also refers to a variable image where its argument is x.

Cropping uses TensorFlow’s sample_distorted_bounding_box with area range 0.08–1 and aspect range 3/4–4/3, followed by bicubic resizing; the ranges are those the appendix gives. The supplement itself has a small inconsistency: Appendix B.3’s text calls Table B.3 “Figure B.3”, a label B.4 also uses for its eigenvalue plot.