MoCo

MoCo (Momentum Contrast) is a self-supervised method for learning image representations with a contrastive loss. It was introduced by Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie and Ross Girshick of Facebook AI Research at CVPR 2020. Two encoders, ResNets in the original paper, map two augmented views of each image to normalized vectors. The loss asks the view’s “query” to find its own “key” among many keys from other images. MoCo’s contribution is how those other keys are kept: in a queue that holds the keys of many recent mini-batches, encoded by a momentum encoder whose weights are a slowly moving average of the query encoder’s. MoCo v2 (2020) changed the augmentation and the projection head. MoCo v3 (2021) dropped the queue and was used to study self-supervised training of Vision Transformers.

The loss is InfoNCE, which has its own article, as does contrastive learning in general. This article covers the mechanism, its ablations, its versions, and a small example of what the momentum encoder does to the keys in a queue.

Problem Addressed

He et al. (2020) describe contrastive learning as training an encoder for dictionary look-up. An encoded query $q$ should match one key $k_+$ in a dictionary of encoded keys $\{k_0, k_1, k_2, \dots\}$ and differ from the rest. Their hypothesis was that such dictionaries should be (i) large, to sample the continuous, high-dimensional space of images well, and (ii) consistent, meaning that all keys are produced by the same or a similar encoder, so that comparing them with a query is meaningful (§1, §3.2).

Two existing mechanisms each gave up one of the two properties (§3.2, Figure 2):

End-to-end. Queries and keys from the current mini-batch are encoded by networks updated by back-propagation. The keys are consistent, but the dictionary is the mini-batch, so its size is limited by GPU memory. In their set-up, the largest mini-batch that a machine with 8 Volta 32 GB GPUs could hold was 1,024, and large-batch optimization was itself an open problem (§4.1).
Memory bank. Wu et al. (2018) stored a representation of every training image and sampled the dictionary from it, without back-propagation through the keys. The dictionary can be large, but each stored representation was last updated when its image was last seen. He et al. argue that the sampled keys therefore come from encoders at many different steps over the past epoch and are less consistent.

The two papers describe the memory bank differently, and each description is attributed here to the version it comes from. MoCo (§3.2) says that Wu et al. adopted a momentum update on the bank’s representations, and its reference list cites an updated arXiv version of Wu et al., which was not read here. The CVPR 2018 version of Wu et al. writes each new feature directly into the bank ( $f_i \to v_i$ ) and stabilizes training with a proximal term $\lambda \lVert v_i^{(t)} - v_i^{(t-1)} \rVert_2^2$ instead (§3.1, §3.3).

MoCo’s authors present it as a mechanism for building dictionaries, not as a new pretext task. They used instance discrimination, following Wu et al.: two random augmentations of the same image form the positive pair, and views of other images are negatives (§3.3). The pretext tasks article covers the pretext tasks that came before.

Architecture

MoCo has two encoders of the same architecture and a queue of keys. The query encoder $f_q$ is trained by gradient descent. The key encoder $f_k$ starts as a copy of $f_q$ and then follows it as a moving average. The queue holds the keys of the most recent mini-batches (Figure 1, Algorithm 1).

Query and Key Encoders

The encoder in MoCo v1 is a standard ResNet whose last fully connected layer, after global average pooling, outputs a 128-dimensional vector, which is then $\ell_2$ -normalized. Queries are $q = f_q(x^q)$ and keys are $k = f_k(x^k)$ (§3.3). The authors stress that no architecture is customized for the pretext task: unlike methods that patchify the input, tailor receptive fields or combine two networks, MoCo’s ResNet-50 can be transferred to downstream tasks unchanged (§4.1). MoCo v3 counts MoCo among the self-supervised frameworks based on Siamese networks (§1); in our terms, the two encoders do not share weights but are tied through the moving average.

Momentum Update

The key encoder receives no gradient. With $\theta_q$ and $\theta_k$ the parameters of $f_q$ and $f_k$ , every step updates

\theta_k \leftarrow m\,\theta_k + (1 - m)\,\theta_q, \qquad m \in [0, 1),

which is Eq. (2) of He et al. (2020). Only $\theta_q$ is updated by back-propagation. A gradient into $f_k$ would have to flow to every key in the queue, which is intractable. The naive alternative, copying $f_q$ into $f_k$ at every step ( $m = 0$ ), gave poor results. The authors attribute this to a rapidly changing key encoder that makes the keys inconsistent. A large momentum, $m = 0.999$ by default, makes $\theta_k$ evolve more smoothly than $\theta_q$ , so keys encoded in different mini-batches come from encoders that differ only a little (§3.2).

This momentum is unrelated to the momentum term of the SGD optimizer that trains $f_q$ . The same moving-average encoder appears as the teacher of DINO, which has no queue and no contrastive loss; that article covers the teacher.

Dictionary Queue

The keys form a queue of $K$ entries ( $C \times K$ in Algorithm 1, with $C = 128$ ). After each step, the keys of the current mini-batch are enqueued and those of the oldest mini-batch are dequeued (§3.2). The queue separates the dictionary size from the mini-batch size, so $K$ becomes a free hyperparameter. Removing the oldest mini-batch first also removes the keys that are most outdated. With the main setting of $K = 65{,}536$ and a mini-batch of 256, the queue holds the keys of the last 256 mini-batches (our arithmetic).

The authors also note that, unlike a memory bank, MoCo keeps no record per sample. It is therefore more memory-efficient and can train on billion-image datasets, which they describe as possibly intractable for a memory bank (§3.2).

Shuffling Batch Normalization

Both encoders use batch normalization (BN), as in a standard ResNet, computed separately on each GPU. With plain per-GPU BN, the model appeared to “cheat” the pretext task. The authors suggest that BN leaks information between the samples of a sub-batch (§3.3). In the supplement (A.9), training without the fix pushed the pretext accuracy of the $(K+1)$ -way look-up above 99.9% quickly, while a k-NN monitor of ImageNet accuracy dropped. The authors’ reading is that the sub-batch statistics act as a signature of the sub-batch that holds the positive key.

Shuffling BN removes that signature. Before the key encoder runs, the sample order of the mini-batch is shuffled across GPUs, and the original order is restored after encoding; the query encoder’s order is not changed. A query and its positive key are then normalized with statistics from different subsets. MoCo and its end-to-end ablation both use shuffling BN. The memory-bank variant does not need it, because its positive keys come from past mini-batches (§3.3, A.9).

Training

Objective

He et al. (2020) use InfoNCE with dot-product similarity (Eq. 1):

\mathcal{L}_q = -\log \frac{\exp(q \cdot k_+ / \tau)}{\sum_{i=0}^{K} \exp(q \cdot k_i / \tau)},

where the sum runs over one positive and the $K$ negatives in the queue, and $\tau = 0.07$ following Wu et al. It is the log loss of a $(K+1)$ -way softmax classifier that tries to classify $q$ as $k_+$ (§3.1). The InfoNCE Loss article covers the loss’s origin in noise-contrastive estimation, its mutual-information bound and its temperature, and Contrastive Learning covers its gradient.

Algorithm 1 of the paper performs one training step as follows:

Augment a mini-batch of $N$ images twice, giving $x^q$ and $x^k$ .
Encode the queries with $f_q$ and the keys with $f_k$ , and detach the keys from the graph.
Compute $N$ positive logits $q \cdot k_+$ and an $N \times K$ matrix of negative logits against the queue, then concatenate them so that the positive is always class 0.
Apply the cross-entropy loss to the logits divided by $\tau$ , and update $f_q$ with SGD.
Update $f_k$ with the momentum rule.
Enqueue the current keys and dequeue the oldest mini-batch.

Data and Recipe

The data augmentation follows Wu et al.: a $224 \times 224$ crop of a randomly resized image, random color jittering, random horizontal flip and random grayscale conversion (§3.3). The paper pre-trains on two datasets (§4):

ImageNet-1M (IN-1M): the 1.28 million training images of ImageNet, used without labels. SGD with weight decay 0.0001 and momentum 0.9, mini-batch 256 on 8 GPUs, initial learning rate 0.03, 200 epochs, with the learning rate multiplied by 0.1 at epochs 120 and 160. This took about 53 hours for ResNet-50.
Instagram-1B (IG-1B): about 940 million public Instagram images from 1,500 hashtags related to ImageNet categories, which the authors describe as relatively uncurated and long-tailed. Mini-batch 1,024 on 64 GPUs, learning rate 0.12 decayed by 0.9× every 62.5k iterations, 1.25 million iterations (about 1.4 epochs). This took about 6 days for ResNet-50.

Ablations

All ablations pre-train ResNet-50 on IN-1M and report the 1-crop top-1 accuracy of a linear classifier on frozen features, on the ImageNet validation set (§4.1).

Mechanisms (Figure 3). With the same pretext task and the same InfoNCE loss, only the way keys are maintained was varied. Accuracy as printed in the figure:

$K$	256	512	1,024	4,096	16,384	65,536
End-to-end	54.9	56.3	57.3	–	–	–
Memory bank	50.0	52.0	54.1	56.5	57.8	58.0
MoCo	54.7	56.4	57.5	59.0	60.4	60.6

All three improved with larger $K$ . End-to-end tracked MoCo while it fit in memory; it needed the linear learning-rate scaling rule, without which accuracy dropped by about 2% at a mini-batch of 1,024. End-to-end has $K - 1$ negatives because the positive key is in the same mini-batch. At $K = 65{,}536$ the memory bank was 2.6 points below MoCo. Its 58.0% comes from the authors’ improved implementation of Wu et al. With NCE and $K = 4{,}096$ , as in Wu et al., they reproduced 54.3%, close to the 54.0% Wu et al. reported (footnote 2).

Momentum ( $K = 4{,}096$ ).

$m$	0	0.9	0.99	0.999	0.9999
Accuracy (%)	fail	55.2	57.8	59.0	58.9

With $m = 0$ the training loss oscillated and did not converge. The authors read the table as showing that a slowly progressing key encoder is beneficial, with good results anywhere from 0.99 to 0.9999.

Shuffling BN. As described above, the supplement’s training curves (Figure A.1) show MoCo overfitting to the pretext task without it, and the authors report the same for the end-to-end variant.

Example: How Old Keys Drift

The example below asks what the momentum update does to the keys in a queue. No network is trained. Instead, a fixed linear encoder $f(x) = Wx / \lVert Wx \rVert$ (64 inputs, 16 outputs) is applied to 512 contrast-normalized $8 \times 8$ patches, 128 from each of four images bundled with scikit-image (camera, coffee, chelsea, rocket, in gray; patches with a raw standard deviation below 0.02 are skipped). The query weights follow a prescribed path that stands in for training:

\theta_q(t) = W_0 + d\,(t - T)\,V + s\,\sigma\,\varepsilon_t, \qquad t = 0, \dots, T = 8192.

Here $V$ is a fixed direction scaled so that 256 steps move the weights by 2% of $\lVert W_0 \rVert$ (Frobenius norm), and $\varepsilon_t$ is fresh standard normal noise per entry with $\sigma^2 = 0.05/64$ . The switches $d, s \in \{0, 1\}$ give three regimes: steady drift only, white noise only, and both. The key encoder starts as a copy of the query encoder, as in Algorithm 1, and follows the momentum rule for $m \in \{0, 0.9, 0.99, 0.999\}$ ; all four runs use the same noise. The measure is the mean over patches of $1 - \cos$ between a key encoded at step $T - a$ and the same patch re-encoded by the current key encoder, for key ages $a \in \{1, 16, 256\}$ . The age 256 matches the oldest key in MoCo’s default queue. The script also prints $1 - \cos$ between the query encoding $\theta_q(T)x$ and the key encoding $\theta_k(T)x$ of the same patch.

Because the encoder is linear and the paths are prescribed, the weight-space behaviour has closed forms. The following derivations are ours, not the paper’s.

Drift only. Suppose $\theta_k(t-1) = \theta_q(t-1) - L_{t-1}V$ . One update gives $\theta_k(t) = \theta_q(t) - m(1 + L_{t-1})V$ , so with $L_0 = 0$ ,

L_t = \frac{m\,(1 - m^t)}{1 - m} \;\longrightarrow\; \frac{m}{1 - m}.

The key encoder is exactly the query encoder of $L_t$ steps earlier. Once it is past its start-up, it moves at the same speed as the query encoder: $\theta_k(T) - \theta_k(T-a) \to aV$ . Steady drift therefore ages queued keys whatever $m$ is.

Noise only. Writing $\theta_k(t) - W_0 = \sigma\,[\,m^t\varepsilon_0 + (1-m)\sum_{s=1}^{t} m^{t-s}\varepsilon_s]$ gives, per weight entry and in units of $\sigma^2$ , the stationary values

\operatorname{Var}\theta_k = \frac{1-m}{1+m}, \qquad \operatorname{Var}\big[\theta_k(t) - \theta_k(t-a)\big] = \frac{2(1-m)(1-m^a)}{1+m}, \qquad \operatorname{Var}\big[\theta_q(t) - \theta_k(t)\big] = \frac{2m^2}{1+m}.

The code compares the measured variances with the exact finite- $T$ versions of these expressions, which include the start-up terms in $m^{2T}$ . The first two expressions shrink as $m \to 1$ , while the third grows: the momentum encoder makes keys agree with each other, but it disagrees more with the current query encoder.

Before the run, a separate script (not shown) fixed the criteria. It computed the drift-only cosines exactly from the closed-form weights. For the noise-driven cosines, it used our own small-angle approximation, $1 - \cos \approx \lVert P_\perp e \rVert^2 / (2\lVert y \rVert^2)$ for a perturbation $e$ of an output $y$ , with $P_\perp$ the projection orthogonal to $y$ , applied to the variances above. The approximation set the expected sizes behind hypotheses H1, H4 and H5 below and plays no part in the printed checks.

The planned checks were:

B0: the data have the planned properties.
C1, C2, C5: the measured variances match the closed forms within four standard errors, $4\sqrt{2/(1024 \cdot 8)} = 0.0625$ relative.
C3: the measured lag matches $L_T$ within $10^{-9}$ steps.
C4: the EMA recursion matches the closed-form drift path within $10^{-10}$ .

The planned hypotheses were:

H1: under noise, staleness falls with $m$ at every age, with no overlap between the seed ranges of neighbouring momenta, and at age 256 it is at least 100 times smaller at $m = 0.999$ than at $m = 0$ .
H2: under drift, staleness relative to $m = 0$ stays within $[0.90, 1.00]$ .
H3: under drift, the query–key mismatch grows with $m$ and at $m = 0.999$ exceeds the age-256 staleness.
H4: with drift and noise at age 256, staleness falls with $m$ ; at $m = 0.999$ it lies within 10% of the drift-only value on average (seed maximum at most 1.15 times); at $m = 0$ it is at least 10 times the drift-only value.
H5: under noise, the query–key mismatch is at least 5 times the age-256 staleness for every $m \geq 0.9$ , and exactly 0 at $m = 0$ .

The script uses NumPy 2.3.5 and scikit-image 0.25.2 (Python 3.13). It runs in about 2 seconds, and two runs gave identical output.

import os
os.environ["OPENBLAS_NUM_THREADS"] = "1"; os.environ["OMP_NUM_THREADS"] = "1"; os.environ["MKL_NUM_THREADS"] = "1"
import numpy as np
from skimage import data, color, img_as_float

T, D, PS = 8192, 16, 8                 # steps, embedding size, patch side
MS = np.array([0.0, 0.9, 0.99, 0.999]) # momentum coefficients
AGES = [1, 16, 256]                    # key age in steps (256 = MoCo's 65,536 keys / batch 256)
SEEDS = range(1, 9)                    # noise seeds
rng = np.random.default_rng(0)

# 512 contrast-normalized 8x8 patches from four bundled images (no flat patches)
imgs = [img_as_float(data.camera())] + [color.rgb2gray(f()) for f in (data.coffee, data.chelsea, data.rocket)]
X, raw_std = [], []
for im in imgs:
    n = 0
    while n < 128:
        r, c = rng.integers(0, im.shape[0] - PS), rng.integers(0, im.shape[1] - PS)
        p = im[r:r + PS, c:c + PS].ravel()
        if p.std() < 0.02:
            continue
        raw_std.append(p.std()); p = p - p.mean(); X.append(p / np.linalg.norm(p)); n += 1
X = np.array(X)

# Query encoder theta_q(t) = W0 + drift*(t - T)*V + noise*SIGMA*eps_t (linear, 64 -> 16)
W0 = rng.normal(0, 1 / PS, (D, PS * PS))
V = rng.normal(size=(D, PS * PS)); V *= 0.02 * np.linalg.norm(W0) / 256 / np.linalg.norm(V)
SIGMA = np.sqrt(0.05 / PS**2)
mm = MS[:, None, None]

def run(drift, noise, seed):
    """Key encoder theta_k <- m theta_k + (1-m) theta_q for all four m at once (same eps)."""
    g = np.random.default_rng(seed)
    q = lambda t: W0 + drift * (t - T) * V + noise * SIGMA * g.standard_normal(W0.shape)
    Wq = q(0); K = np.repeat(Wq[None], len(MS), axis=0)   # f_k.params = f_q.params
    snap = {}
    for t in range(1, T + 1):
        Wq = q(t); K = mm * K + (1 - mm) * Wq
        if T - t in AGES:
            snap[T - t] = K.copy()
    snap[0] = K
    return snap, Wq

def one_minus_cos(Wa, Wb):
    ya, yb = X @ Wa.T, X @ Wb.T
    ya /= np.linalg.norm(ya, axis=1, keepdims=True); yb /= np.linalg.norm(yb, axis=1, keepdims=True)
    return 1 - np.mean(np.sum(ya * yb, axis=1))

def stale(snap, Wq):  # rows m; columns ages 1, 16, 256, then query-key
    return np.array([[one_minus_cos(snap[a][i], snap[0][i]) for a in AGES] + [one_minus_cos(Wq, snap[0][i])]
                     for i in range(len(MS))])

print("B0 data: %d patches, max |mean| %.1e, max |norm-1| %.1e, min raw std %.3f; ||V||*256/||W0|| %.3f, p*SIGMA^2 %.3f"
      % (len(X), np.abs(X.mean(1)).max(), np.abs(np.linalg.norm(X, axis=1) - 1).max(), min(raw_std),
         np.linalg.norm(V) * 256 / np.linalg.norm(W0), PS**2 * SIGMA**2))

# ---- drift only (deterministic)
snap, Wq = run(1, 0, 0)
Sd = stale(snap, Wq)
print("\nDRIFT ONLY      lag (steps)                       1 - cos: key age 1        16       256   query-key")
for i, m in enumerate(MS):
    lag_hat = np.sum((Wq - snap[0][i]) * V) / np.sum(V * V)
    lag_T = m * (1 - m**T) / (1 - m)
    resid = np.linalg.norm(Wq - snap[0][i] - lag_hat * V)
    print("m=%-6g  %9.3f (exact %9.3f, m/(1-m) %6.0f) |err| %.1e  %.4e %.4e %.4e %.4e"
          % (m, lag_hat, lag_T, m / (1 - m), max(abs(lag_hat - lag_T), resid), *Sd[i]))
cf_err = max(np.abs(snap[a][i] - (W0 - a * V - V * MS[i] * (1 - MS[i]**(T - a)) / (1 - MS[i]))).max()
             for a in [0] + AGES for i in range(len(MS)))
print("C4 max |recursion - closed form| over m and ages: %.1e" % cf_err)
print("H2 staleness ratio m/m=0 at ages 1,16,256:", " ".join("m=%g: %s" % (m, " ".join("%.4f" % r for r in Sd[i, :3] / Sd[0, :3]))
                                                    for i, m in enumerate(MS) if i))

# ---- white noise only and drift + noise (seeds 1-8, common random numbers across m)
def f_var(m):    return m**(2 * T) + (1 - m) * (1 - m**(2 * T)) / (1 + m)
def f_diff(m, a):
    return ((1 - m**a)**2 * (m**(2 * (T - a)) + (1 - m)**2 * (1 - m**(2 * (T - a))) / (1 - m**2))
            + (1 - m)**2 * (1 - m**(2 * a)) / (1 - m**2))
def f_qk(m):     return m**2 + (1 - m)**2 * m**2 * (1 - m**(2 * (T - 1))) / (1 - m**2) + m**(2 * T)
SE = np.sqrt(2 / (W0.size * len(SEEDS)))
res = {}
for name, drift in (("NOISE ONLY", 0), ("DRIFT + NOISE", 1)):
    S, R = [], []
    for s in SEEDS:
        snap, Wq = run(drift, 1, s)
        S.append(stale(snap, Wq))
        if not drift:   # ratios of measured to exact variances (per entry, units of SIGMA^2)
            R.append([[np.mean((snap[0][i] - W0)**2)] + [np.mean((snap[0][i] - snap[a][i])**2) for a in AGES]
                      + [np.mean((Wq - snap[0][i])**2)] for i in range(len(MS))])
    S = np.array(S); res[name] = S
    print("\n%s (seeds 1-8)   1 - cos, mean [min, max]: key age 1, 16, 256, query-key" % name)
    for i, m in enumerate(MS):
        print("m=%-6g " % m + "  ".join("%.3e [%.3e, %.3e]" % (S[:, i, j].mean(), S[:, i, j].min(), S[:, i, j].max())
                                       for j in range(4)))
    if not drift:
        R = np.array(R).sum(0) / len(SEEDS) / SIGMA**2
        print("C1/C2/C5 measured/exact (tol |r-1| <= 4 SE = %.4f): Var(theta_k), diff at ages 1, 16, 256, query-key" % (4 * SE))
        for i, m in enumerate(MS):
            ex = [f_var(m)] + [f_diff(m, a) for a in AGES] + [f_qk(m)]
            print("m=%-6g " % m + " ".join("%.4f" % (r / e) if e > 0 else "(exact 0, measured %.1e)" % r
                                           for r, e in zip(R[i], ex)))
Sn, Sb = res["NOISE ONLY"], res["DRIFT + NOISE"]
print("\nH1 noise, age 256: min over seeds at m=0 / max over seeds at m=0.999 = %.1f" % (Sn[:, 0, 2].min() / Sn[:, 3, 2].max()))
print("H4 drift+noise / drift only at age 256, mean [min, max]:",
      " ".join("m=%g %.3f [%.3f, %.3f]" % (m, *(lambda r: (r.mean(), r.min(), r.max()))(Sb[:, i, 2] / Sd[i, 2]))
               for i, m in enumerate(MS)))
print("H5 noise: query-key / age-256 staleness, mean [min over seeds]:",
      " ".join("m=%g %.1f [%.1f]" % (m, (Sn[:, i, 3] / Sn[:, i, 2]).mean(), (Sn[:, i, 3] / Sn[:, i, 2]).min())
               for i, m in enumerate(MS) if i))

B0 data: 512 patches, max |mean| 5.7e-16, max |norm-1| 2.2e-16, min raw std 0.020; ||V||*256/||W0|| 0.020, p*SIGMA^2 0.050

DRIFT ONLY      lag (steps)                       1 - cos: key age 1        16       256   query-key
m=0           0.000 (exact     0.000, m/(1-m)      0) |err| 0.0e+00  3.0216e-09 7.7328e-07 1.9695e-04 0.0000e+00
m=0.9         9.000 (exact     9.000, m/(1-m)      9) |err| 1.7e-13  3.0205e-09 7.7300e-07 1.9687e-04 2.4471e-07
m=0.99       99.000 (exact    99.000, m/(1-m)     99) |err| 7.1e-13  3.0092e-09 7.7010e-07 1.9608e-04 2.9554e-05
m=0.999     998.725 (exact   998.725, m/(1-m)    999) |err| 1.0e-12  2.8690e-09 7.3412e-07 1.8648e-04 2.9407e-03
C4 max |recursion - closed form| over m and ages: 2.6e-15
H2 staleness ratio m/m=0 at ages 1,16,256: m=0.9: 0.9996 0.9996 0.9996 m=0.99: 0.9959 0.9959 0.9956 m=0.999: 0.9495 0.9494 0.9469

NOISE ONLY (seeds 1-8)   1 - cos, mean [min, max]: key age 1, 16, 256, query-key
m=0      4.343e-02 [3.684e-02, 4.670e-02]  4.343e-02 [4.117e-02, 4.653e-02]  4.465e-02 [3.654e-02, 4.805e-02]  0.000e+00 [0.000e+00, 0.000e+00]
m=0.9    2.386e-04 [2.080e-04, 2.588e-04]  1.948e-03 [1.610e-03, 2.299e-03]  2.520e-03 [2.080e-03, 2.827e-03]  1.862e-02 [1.610e-02, 2.052e-02]
m=0.99   2.243e-06 [1.992e-06, 2.423e-06]  3.352e-05 [2.888e-05, 4.059e-05]  2.175e-04 [1.913e-04, 2.437e-04]  2.117e-02 [1.852e-02, 2.306e-02]
m=0.999  2.233e-08 [1.997e-08, 2.437e-08]  3.562e-07 [2.973e-07, 4.460e-07]  5.301e-06 [4.897e-06, 6.299e-06]  2.145e-02 [1.886e-02, 2.314e-02]
C1/C2/C5 measured/exact (tol |r-1| <= 4 SE = 0.0625): Var(theta_k), diff at ages 1, 16, 256, query-key
m=0      0.9891 1.0008 1.0043 0.9679 (exact 0, measured 0.0e+00)
m=0.9    0.9807 0.9990 0.9882 0.9881 0.9990
m=0.99   1.0137 0.9902 0.9811 0.9955 0.9902
m=0.999  1.0148 0.9891 0.9805 1.0105 0.9891

DRIFT + NOISE (seeds 1-8)   1 - cos, mean [min, max]: key age 1, 16, 256, query-key
m=0      4.343e-02 [3.685e-02, 4.670e-02]  4.343e-02 [4.115e-02, 4.653e-02]  4.453e-02 [3.603e-02, 4.809e-02]  0.000e+00 [0.000e+00, 0.000e+00]
m=0.9    2.386e-04 [2.080e-04, 2.589e-04]  1.947e-03 [1.607e-03, 2.280e-03]  2.655e-03 [2.073e-03, 3.021e-03]  1.862e-02 [1.610e-02, 2.052e-02]
m=0.99   2.239e-06 [1.984e-06, 2.428e-06]  3.387e-05 [2.952e-05, 3.848e-05]  4.041e-04 [3.605e-04, 4.880e-04]  2.117e-02 [1.847e-02, 2.313e-02]
m=0.999  2.434e-08 [2.152e-08, 2.718e-08]  1.038e-06 [8.841e-07, 1.199e-06]  1.893e-04 [1.857e-04, 1.942e-04]  2.396e-02 [2.084e-02, 2.638e-02]

H1 noise, age 256: min over seeds at m=0 / max over seeds at m=0.999 = 5800.7
H4 drift+noise / drift only at age 256, mean [min, max]: m=0 226.104 [182.956, 244.170] m=0.9 13.487 [10.530, 15.346] m=0.99 2.061 [1.838, 2.489] m=0.999 1.015 [0.996, 1.042]
H5 noise: query-key / age-256 staleness, mean [min over seeds]: m=0.9 7.5 [6.1] m=0.99 97.9 [79.0] m=0.999 4060.3 [3592.0]

All checks passed, and all five hypotheses held as planned.

Checks.

B0: the patches have zero mean and unit norm to rounding. The minimum raw standard deviation prints 0.020, exactly at its limit; it holds by construction, because the code rejects any patch below 0.02.
C3 and C4: the drift-only lag and path agree with the closed forms to rounding.
C1, C2 and C5: in the noise regime, every variance ratio is within the 0.0625 tolerance for every $m$ and age; the largest deviation is $m = 0$ at age 256. The query–key difference at $m = 0$ is exactly zero.

Drift only. Staleness is set by the drift, not by the momentum. Relative to $m = 0$ , every ratio lies between 0.9469 and 0.9996, inside H2’s band. The age-256 staleness of about $2 \times 10^{-4}$ falls only slightly as $m$ rises. In our reading, the small decrease at $m = 0.999$ comes from the lag: the key encoder there is the query encoder of about 999 steps earlier, a different point on the path. The lag column shows that point directly: 9, 99 and 998.7 steps, against $m/(1-m)$ = 9, 99 and 999, with the last still short of its limit after 8,192 steps.

H3 held. The query–key mismatch rises from 0 to $2.9 \times 10^{-3}$ at $m = 0.999$ , 16 times the age-256 staleness at that momentum. With MoCo’s default momentum, the key encoder trails the query encoder by more steps than the oldest key in the default queue has waited.

Noise only. H1 held at every age. The means fall by orders of magnitude from $m = 0$ to $m = 0.999$ , the seed ranges of neighbouring momenta do not overlap, and at age 256 the smallest $m = 0$ seed is 5,800 times the largest $m = 0.999$ seed. At $m = 0.9$ , staleness at age 16 is already more than three quarters of its age-256 value; at $m = 0.999$ it grows about fifteenfold from age 16 to 256, close to the ratio of the factors $1 - m^a$ at the two ages, about 14 (our arithmetic).

H5 held. For every $m \geq 0.9$ , the query–key mismatch (about $2 \times 10^{-2}$ ) is far larger than the queue staleness: the smallest seed ratios are 6.1, 79 and 3,592. In this model, the momentum encoder makes all keys, positive and negative, consistent with each other, which is the property MoCo’s argument requires. It does not make them consistent with the query encoder.

Drift and noise. H4 held. At age 256, staleness falls with $m$ from $4.5 \times 10^{-2}$ to $1.9 \times 10^{-4}$ . At $m = 0.999$ it is within 5% of the drift-only floor (ratios 0.996–1.042), while at $m = 0$ it is 183–244 times that floor. In this model, momentum removes the noise component of key staleness and leaves the drift component, which only a shorter queue or slower learning would reduce. How closely real SGD trajectories resemble drift plus white noise is not something this example tests.

Inference

Downstream, the pre-trained encoder is used as a frozen feature extractor or as an initialization. Two evaluation routes appear in the paper:

Frozen features. A supervised linear classifier, a fully connected layer followed by softmax, is trained for 100 epochs on the global-average-pooled ResNet features (§4.1). A grid search put the best initial learning rate at 30 with weight decay 0. The authors read this as a sign that the feature distributions differ substantially from those of supervised ImageNet training. The evaluating representations article covers the protocol.
Fine-tuning. For detection and segmentation, the whole network is fine-tuned. Because the feature magnitudes differ, the authors fine-tune with BN layers that are trained and synchronized across GPUs, rather than frozen. They also add BN to the newly initialized layers, such as the FPN, and they use the supervised baseline’s hyperparameters and schedules unchanged (§4.2).

Performance

Linear classification on ImageNet (Table 1). With $K = 65{,}536$ and $m = 0.999$ , MoCo reached 60.6% top-1 with ResNet-50 (24M parameters in the feature extractor), 63.9% with ResNeXt-50, 65.4% with a 2× wider ResNet-50 and 68.6% with a 4× wider one. In the same table, instance discrimination (Wu et al.) had 54.0% and LocalAgg 58.8% with ResNet-50; CPC v2 had 65.9% with a wider ResNet-170 (303M); and CMC had 64.1% with a two-network ResNet-50 (47M). CMC’s entry is marked as pre-trained with FastAutoAugment, which is supervised by ImageNet labels. The authors claimed the best accuracy among models of similar size, about 24M parameters.

Transfer to detection and segmentation. This was the paper’s main claim: MoCo outperformed ImageNet-supervised pre-training on seven detection or segmentation tasks (§4.2.3). Examples, each fine-tuned with the same settings as the supervised baseline:

PASCAL VOC detection. Faster R-CNN with an R50-C4 backbone, fine-tuned on trainval07+12 and evaluated on test2007, averaged over 5 trials (Table 2b). MoCo IN-1M reached 81.5 AP50, 55.9 AP and 62.6 AP75; MoCo IG-1B reached 82.2, 57.2 and 63.7; supervised IN-1M reached 81.3, 53.5 and 58.8. With the R50-dilated-C5 backbone (Table 2a), MoCo IN-1M was 0.3 AP50 below supervised (81.1 against 81.4) and above it in AP and AP75. The authors note that the transfer advantage depends on the detector structure.
The three mechanisms on VOC (Table 3). Fine-tuned in the same way, the best end-to-end and memory-bank models were below MoCo in every metric for both backbones.
COCO detection and instance segmentation. Mask R-CNN R50-FPN, trained on train2017 and evaluated on val2017 (Table 5). With the 1× schedule, MoCo IN-1M was slightly below supervised (38.5 against 38.9 box AP); with 2×, it was slightly above (40.8 against 40.6), and IG-1B reached 41.1. The authors note that all models are under-trained at 1×.
Other tasks (Table 6).
- COCO keypoint detection: 66.8 AP (IN-1M) and 66.9 (IG-1B), against 65.8 supervised and 65.9 from random initialization.
- COCO dense pose: up to 3.7 points higher in $\text{AP}^{\text{dp}}_{75}$ .
- LVIS v0.5: IG-1B 24.9 mask AP, against 24.4 for a supervised baseline fine-tuned with frozen BN.
- Cityscapes semantic segmentation: up to 0.9 mIoU higher.
- The exceptions: Cityscapes instance segmentation was on par, and VOC semantic segmentation was worse, at 72.5 (IN-1M) and 73.6 (IG-1B) mIoU against 74.4.

In every task, MoCo pre-trained on IG-1B was better than MoCo pre-trained on IN-1M. The authors call this improvement consistently noticeable but relatively small (§4.2.3, §5).

Strengths

Many negatives at a small batch. The queue gives 65,536 negatives with a mini-batch of 256 on one 8-GPU machine, with no large-batch optimizer.
Consistent keys. The momentum encoder keeps the dictionary consistent without any gradient into the key encoder, and the authors call the extra computation of maintaining the queue manageable.
Standard backbone. An unmodified ResNet transfers to detection and segmentation without architectural changes, which made MoCo a fair test of self-supervised pre-training for dense tasks.
Scale. The paper trains on almost a billion relatively uncurated images, a setting the authors expected a memory bank to find hard.

Limitations

Batch-statistics leakage. BN lets the model cheat the pretext task, and the fix, shuffling BN, as described relies on per-GPU BN across several GPUs (§3.3).
Hyperparameters. MoCo adds the queue size $K$ and the momentum $m$ to the temperature $\tau$ . $m = 0$ fails outright and $m = 0.9$ loses 3.8 points against 0.999 (55.2 against 59.0, §4.1). MoCo v3 later found a different optimum ( $m = 0.99$ ) for ViT-B (§6.2).
Linear accuracy. The 60.6% of v1 trailed contemporaneous methods that used bigger or customized networks, and the linear classifier needed an unusual learning rate of 30.
Small gains from more data. The gain from IN-1M to IG-1B was small. The authors suggest that the larger data were not fully exploited and that a more advanced pretext task might help (§5).
A negative case in transfer. VOC semantic segmentation transferred worse than supervised pre-training (Table 6).
False negatives. Instance discrimination treats every other image as a negative. The contrastive learning article discusses the false negatives this creates.

Historical Importance

MoCo was published at CVPR 2020. On the strength of its detection and segmentation results, the authors argued that the gap between unsupervised and supervised representation learning had been “largely closed” in many vision tasks. The representation learning article cites this claim. In our reading, MoCo also offered a second way to get many negatives, alongside the large batches of SimCLR: a short-lived queue of keys from a slowly moving encoder.

The momentum encoder outlived the queue. MoCo v3 notes that SimCLR v2 and BYOL both use a momentum encoder (§6.3). DINO uses one as its teacher, and MoCo v3 itself kept it after dropping the queue.

Variants

Memory Bank (Wu et al., 2018)

MoCo’s main predecessor treats each image as its own class. In the CVPR version (§3), a non-parametric softmax compares a 128-dimensional, $\ell_2$ -normalized feature with stored features $v_j$ of every training image at $\tau = 0.07$ . The softmax over all training images is approximated with noise-contrastive estimation using $m$ noise samples from a uniform noise distribution; this $m$ is a sample count, not MoCo’s momentum. The bank is updated by writing the new feature into the image’s entry, with the proximal term described under Problem Addressed. In Table 1 of that version, on CIFAR-10, NCE approached the full non-parametric softmax as the number of noise samples grew to 4,096. The InfoNCE loss article covers NCE itself.

MoCo v2 (2020)

The MoCo v2 technical report (Chen, Fan, Girshick & He, 2020) was not read here, so it is described only through other sources. MoCo v1 (§4.1) describes it as an extension that reached 71.1% linear accuracy with ResNet-50, up from 60.6%, through small changes to the data augmentation and the output projection head, for which it cites SimCLR. MoCo v3 (§3) lists MoCo v2 at 71.1% after 800 epochs, an improved “MoCo v2+” at 72.2%, and takes its default $\tau = 0.2$ from MoCo v2. As the contrastive learning article reports, the changes were an MLP head, blur augmentation and a cosine learning-rate schedule, and 200 epochs at batch 256 gave 67.5%.

MoCo v3 (2021)

Chen, Xie and He (2021) describe MoCo v3 as an incremental improvement of v1 and v2, introduced to study self-supervised Vision Transformers. The changes (§3, §5):

No queue. Keys come from the same batch, a batch of 4,096, because the queue had diminishing gains once the batch was large enough.
Symmetrized loss. Both views serve as queries: $\text{ctr}(q_1, k_2) + \text{ctr}(q_2, k_1)$ , each a batch-wise InfoNCE scaled by $2\tau$ , with $\tau = 0.2$ .
Prediction head. $f_q$ is backbone + projection head + prediction head, following BYOL; $f_k$ has the backbone and projection head only and is the moving average of the matching parts of $f_q$ . The projection head is a 3-layer MLP and the predictor a 2-layer MLP, with 4,096-dimensional hidden layers, 256-dimensional outputs and BN in every layer.
Optimizer. AdamW at batch 4,096, a 40-epoch warm-up and a cosine schedule.

With ResNet-50 after 800 epochs, linear accuracy rose from 71.1 (v2) and 72.2 (v2+) to 73.8. The authors attribute the gain mainly to the prediction head and the large batch.

Instability. The paper’s central finding is that self-supervised ViT training can be unstable without failing visibly. All runs below are ViT-B/16 with 100 epochs on ImageNet:

Batch size (AdamW, learning rate 1.0e-4): batches of 1,024, 2,048, 4,096 and 6,144 gave 71.5, 72.6, 72.2 and 69.7%. The larger batches showed “dips” in the k-NN curve (Figure 1).
Gradient spikes: the dips followed spikes that appear first in the patch-projection layer (Figure 4).
Random patch projection: freezing that layer at its random initialization raised accuracy at learning rates 0.5e-4, 1.0e-4 and 1.5e-4 from 70.4, 72.2 and 71.7 to 70.8, 72.8 and 73.4 (Figure 5), and also helped SimCLR and BYOL.

The authors write that the trick alleviates the instability but does not solve it.

ViT-B ablations (300 epochs, linear probing):

Ablation	Results (%)
Momentum $m$ = 0, 0.9, 0.99, 0.999	74.3, 75.6, 76.5, 75.0
Prediction head (with, without)	76.5, 75.5
BN in the MLP heads (with, without)	76.5, 74.4 (without needs batch 2,048 to converge)
Position embedding (sin-cos, learned, none)	76.5, 76.1, 74.9

The best momentum here is 0.99, unlike v1’s 0.999 for ResNet-50 with a queue. The authors compare $m = 0$ to SimCLR plus a prediction head and a stop-gradient on the keys; its 74.3% is similar to SimCLR’s 73.9%, and the momentum encoder adds 2.2 points.

Results. In the framework comparison (Table 4, two $224 \times 224$ crops), MoCo v3 reached 76.5% with ViT-B, against 73.9% for SimCLR and BYOL and 71.6% for SwAV. Linear probing results in Table 1 are:

76.7% (ViT-B), 77.6% (ViT-L) and 78.1% (ViT-H);
79.1% with ViT-BN-H, where BN replaces LayerNorm in the MLP blocks;
81.0% with ViT-BN-L/7, which uses $7 \times 7$ patches.

The ViT-B figure of 76.7% equals the 600-epoch entry of the training-length ablation, while the ablations above use 300 epochs. End-to-end fine-tuning on ImageNet-1k gave 83.2% (ViT-B) and 84.1% (ViT-L) (Table 1). The Masked Autoencoder article compares MAE with these numbers. In transfer to CIFAR-10, CIFAR-100, Oxford Flowers-102 and Oxford-IIIT Pets, the self-supervised ViTs beat their ImageNet-supervised counterparts on three of the four datasets (Table 6).

Implementations

The papers give two official repositories, facebookresearch/moco (v1 and v2) and facebookresearch/moco-v3. As of November 2026:

facebookresearch/moco: the GitHub API reported no licence for the repository and listed no files at its root, so nothing from it was read.
facebookresearch/moco-v3: licensed under Creative Commons Attribution-NonCommercial 4.0, which excludes commercial use. Only the licence was read.

Algorithm 1 of each paper gives PyTorch-like pseudocode for one training step. The example above implements only the momentum update, $\theta_k \leftarrow m\theta_k + (1-m)\theta_q$ , and the queue ages, not a trained model.