Emerging Properties in Self-Supervised Vision Transformers

“Emerging Properties in Self-Supervised Vision Transformers” was written by Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski and Armand Joulin, of Facebook AI Research, Inria and Sorbonne University. It appeared on arXiv on 29 April 2021 and was published at the IEEE/CVF International Conference on Computer Vision (ICCV) in 2021. The paper asked whether self-supervised training gives Vision Transformers properties that supervised ViTs and convolutional networks lack, reported two such properties, and packaged the training recipe that produced them as DINO, which the authors interpret as “self-distillation with no labels”. The method itself, its collapse analysis and its successors DINOv2 and DINOv3 are explained in the DINO article; this article covers the 2021 publication: what it set out to show, what it measured, and how its claims hold up against its own tables.

Versions. This article was checked against four copies: the ICCV paper and its supplementary material on the CVF open-access site, and arXiv v1 (29 April 2021) and v2 (24 May 2021). Between v1 and v2, every number is unchanged; v2 renames the small model from DeiT-S to ViT-S, adds references, rewords parts of the related work and adds a paragraph relating DINO to unsupervised image classification (UIC) in Appendix B. The ICCV version differs more. Its eight-page main paper keeps Tables 1–7, Figures 1–5 and Equations (1)–(4) with the same numbers as arXiv, but arXiv’s §5.2–5.5 (teacher choice, collapse, compute, small batches) move to Appendix D of the supplement, and the supplement numbers its tables and figures from 1. The ICCV conclusion also lacks two sentences of the arXiv conclusion, discussed under Impact, and its supplement lacks arXiv’s Figure 8 (attention from chosen reference points), the ResNet-50 teacher-versus-student curve in Appendix D and v2’s UIC paragraph. The results themselves are identical in all copies. Where numbering differs, both are given:

Content	arXiv v1 and v2	ICCV supplement
Teacher variants, training dynamics	§5.2, Figure 6	Appendix D, Figure 2
Collapse study	§5.3, Equation (5), Figure 7	Appendix D, Equation (1), Figure 3
Time and memory	§5.4, Table 8	Appendix D, Table 7
Batch size	§5.5, Table 9	Appendix D, Table 8
k-NN and linear on seven benchmarks	Appendix A, Table 10	Appendix A, Table 1
Pre-training a supervised ViT-B/16	Table 11	Table 2
Low-shot ImageNet	Table 12	Table 3
Comparison with MoCo-v2, BYOL and SwAV	Tables 13–15	Tables 4–6
Projection head, attention maps, t-SNE	Figures 9, 10, 11	Figures 1, 4, 5

Problem

In early 2021, Vision Transformers had just been shown to compete with convolutional networks (Dosovitskiy et al., 2021), and DeiT had trained them on ImageNet-1k alone (Touvron et al., 2021). The authors’ assessment was that ViTs had not yet delivered clear benefits over convnets: they cost more compute, needed more data, and their features showed no unique properties (§1). Their hypothesis was that supervision was to blame. In language, transformers had succeeded through self-supervised pretext tasks such as BERT’s masked words, which give a richer signal than one label per sentence; an image label likewise reduces an image to one of a few thousand predefined concepts.

Self-supervised learning on convnets had meanwhile caught up with supervised pre-training. Methods differed mainly in how they avoided collapse: negatives in instance discrimination and contrastive learning, balanced clustering in SwAV (Caron et al., 2020), or BYOL’s predictor with a momentum target (Grill et al., 2020). The paper set out to test such components on ViTs and to keep only those that mattered. Chen, Xie and He’s “An Empirical Study of Training Self-Supervised Vision Transformers” (MoCo v3), posted on arXiv on 5 April 2021, was published at the same ICCV (Chen et al., 2021).

Contribution

• Two properties of self-supervised ViTs. The last-layer self-attention of the class token contains the scene layout and object boundaries (Figure 1), and the frozen features work as a k-nearest-neighbor classifier, reaching 78.3% top-1 on ImageNet with a small ViT.
• Which components matter. Good k-NN accuracy appeared only when a momentum encoder and multi-crop augmentation were combined, and smaller patches improved the features (§1, Table 7, Figure 5).
• DINO. A student predicts a momentum teacher’s output with a cross-entropy loss, and collapse is avoided by centering and sharpening the teacher output alone, without a predictor or negatives; with ViTs the system uses no batch normalization at all. The same recipe works for ViTs and ResNet-50.
• Benchmarks. 80.1% linear accuracy on ImageNet with ViT-B/8, and parity with the best self-supervised ResNet-50.
• Frozen-feature transfer to image retrieval, copy detection, video object segmentation and attention-based masks, and better fine-tuning transfer than supervised pre-training.
• Modest compute. In the authors’ words, two 8-GPU servers for 3 days reach 76.1% linear accuracy with ViT-S/16 (§1).
• Code and models, released as facebookresearch/dino under the Apache License 2.0. As of October 2026 GitHub marks the repository as archived, and its README points to DINOv2.

Method

The DINO article explains the objective, the momentum teacher and centering; this section records how the paper states them. The paper presents DINO through knowledge distillation (§3.1): a student $g_{\theta_s}$ and a teacher $g_{\theta_t}$ of the same architecture each turn an image into a softmax over $K$ dimensions, with temperatures $\tau_s$ and $\tau_t$, and the student minimizes the cross-entropy to the teacher (Equation 2). Algorithm 1 gives PyTorch-style pseudocode without multi-crop: two augmented views $x_1, x_2$, the symmetric loss $H(t_1, s_2)/2 + H(t_2, s_1)/2$, an exponential moving average of the student into the teacher, and an update of the center $C$ from the teacher outputs. Equation (3) extends the loss to a set of views: two global views at $224^2$ pixels, “for example” covering more than 50% of the image, and several local views at $96^2$ covering less. All views pass through the student and only the global ones through the teacher, which the paper calls “local-to-global” correspondence. Appendix E gives the actual scale ranges, $(s, 1)$ for global and $(0.05, s)$ for local crops, with $s$ around 0.3 best.

The teacher is updated as $\theta_t \leftarrow \lambda \theta_t + (1 - \lambda)\theta_s$ with $\lambda$ rising from 0.996 to 1 on a cosine schedule. The paper relates this momentum encoder less to MoCo’s queue than to the mean teacher of semi-supervised learning and to Polyak–Ruppert averaging. The head is a three-layer MLP with hidden width 2,048, an $\ell_2$-normalized bottleneck and a weight-normalized layer to $K$ outputs, with $K = 65{,}536$ and a 256-dimensional bottleneck by default (Appendix C). Centering adds a bias $c$ to the teacher output, updated by Equation (4); sharpening is the low teacher temperature.

Training. Models were pre-trained on ImageNet without labels with AdamW, batch size 1,024 over 16 GPUs for ViT-S/16, a learning rate of $0.0005 \times \text{batch size}/256$ reached after a 10-epoch linear warm-up and then decayed with a cosine schedule, weight decay rising from 0.04 to 0.4, $\tau_s = 0.1$, and $\tau_t$ warmed linearly from 0.04 to 0.07 over the first 30 epochs (§3.2). Augmentations follow BYOL (color jittering, Gaussian blur, solarization), and position embeddings are interpolated bicubically to the crop sizes. The networks follow DeiT’s implementation; Table 1 lists them: ViT-S (21M parameters, 6 heads) and ViT-B (85M, 12 heads), each with $16 \times 16$ or $8 \times 8$ patches, and ResNet-50 (23M). At $224^2$, the /8 models process 785 tokens against 197, and their V100 throughput falls from 1,007 to 180 images per second for ViT-S and from 312 to 63 for ViT-B. In Table 2’s ViT-S comparison, DINO, MoCo-v2 and SwAV were trained for 800 epochs and BYOL for 300, because BYOL on ViT-S got worse when trained longer (Appendix D).

Evaluation. Three protocols were used (§3.2, Appendix F):

• Linear. A linear classifier on frozen features, trained with SGD at batch size 1,024 for 100 epochs without weight decay, with the learning rate swept per model, random resized crops and flips in training, and a central crop at test. For ViT-S the features are the [CLS] tokens of the last four layers concatenated, which raised ViT-S/16 from 76.1% (last layer only) to 77.0%; for ViT-B, the last [CLS] token concatenated with average-pooled patch tokens, 78.2% against 78.0%.
• Weighted k-NN, following Wu et al. (2018): the 20 nearest stored training features vote with weight $\exp(T_i \cdot x/0.07)$ (Appendix F), a dot product that the released code takes between $\ell_2$-normalized features, that is, a cosine similarity. The authors added it because linear and fine-tuning results varied widely with hyperparameters such as the learning rate, whereas k-NN needs no tuning, no augmentation and one pass over the data.
• Fine-tuning with DeiT’s protocol, for transfer.

Results

All ImageNet numbers are top-1 accuracy on the validation set, as reported by the authors.

ImageNet Linear and k-NN (Table 2)

The top panel compares methods on the same architecture, the bottom panel the best results across architectures. Selected rows (parameters in millions, throughput in images per second on a V100; an asterisk marks baselines the authors ran themselves):

Method	Architecture	Params	im/s	Linear	k-NN
Supervised	RN50	23	1,237	79.3	79.3
BYOL	RN50	23	1,237	74.4	64.8
SwAV	RN50	23	1,237	75.3	65.7
DINO	RN50	23	1,237	75.3	67.5
Supervised	ViT-S	21	1,007	79.8	79.8
BYOL*	ViT-S	21	1,007	71.4	66.6
MoCo-v2*	ViT-S	21	1,007	72.7	64.4
SwAV*	ViT-S	21	1,007	73.5	66.3
DINO	ViT-S	21	1,007	77.0	74.5
DINO	ViT-B/16	85	312	78.2	76.1
BYOL	RN200w2	250	123	79.6	73.9
DINO	ViT-S/8	21	180	79.7	78.3
SimCLRv2	RN152w3+SK	794	46	79.8	73.1
DINO	ViT-B/8	85	63	80.1	77.4

On ResNet-50 DINO tied SwAV’s linear accuracy and had the highest k-NN accuracy among the self-supervised methods. On ViT-S it led the best baseline by 3.5 points linear (SwAV) and 7.9 points k-NN (BYOL). The authors’ central observation is the k-NN column: 74.5% against 77.0% linear for DINO ViT-S, a closeness they found only with DINO on ViTs, not with the other methods or with ResNet-50 (§4.1). Halving the patch size helped more than going from ViT-S to ViT-B, and the paper’s ViT-B/8 headline of 80.1% came, per the text, with ten times fewer parameters and 1.4 times the throughput of SimCLRv2’s RN152w3+SK. From the table, the ratios are 9.3 and 1.37, and the accuracy margin is 0.3 points.

Appendix A (arXiv Table 10; ICCV supplement Table 1) repeats the comparison with a logistic regression and a k-NN classifier on seven benchmarks, from ImageNet subsets to Places205, VOC07 and Flowers. ViT-S beat ResNet-50 by 2.4 points on average with the logistic classifier and by 5.6 with k-NN, and by 14.1 k-NN points with 1% of ImageNet labels. A logistic regression on frozen ViT-S/16 features with 1% and 10% of ImageNet labels reached 64.5% and 72.2%, which the authors place on par with state-of-the-art semi-supervised models of similar size and speed, without fine-tuning or augmentation (arXiv Table 12; supplement Table 3).

Retrieval and Copy Detection (Tables 3–4)

For landmark retrieval on revisited Oxford and Paris, frozen features were compared by nearest-neighbor search and scored by mean average precision on the Medium (M) and Hard (H) splits. The authors also trained DINO on the 1.2 million clean images of Google Landmarks v2 (GLDv2), without its labels:

Features	Pre-training data	ROxford M	ROxford H	RParis M	RParis H
Supervised RN101+R-MAC (best off-the-shelf reference)	ImageNet	49.8	18.5	74.0	52.1
Supervised ViT-S/16	ImageNet	33.5	8.9	63.0	37.2
DINO ResNet-50	ImageNet	35.4	11.1	55.9	27.5
DINO ViT-S/16	ImageNet	41.8	13.7	63.1	34.4
DINO ViT-S/16	GLDv2	51.5	24.3	75.3	51.6

The text says DINO features outperform those trained on ImageNet with labels; in the table, DINO ViT-S/16 beats its supervised counterpart on three of the four splits but not on RParis Hard (34.4 against 37.2). Trained on GLDv2, it beat the R-MAC reference on all splits except RParis Hard (51.6 against 52.1).

Copy detection used the “strong” subset of INRIA Copydays, whose images are distorted by blur, insertions, print-and-scan and the like, with 10,000 distractor images from YFCC100M. The descriptor concatenates the output [CLS] token with GeM-pooled patch tokens (1,536 dimensions for ViT-B), whitened with a transform learned on 20,000 further YFCC100M images, and compared by cosine similarity. Mean average precision:

Method	Architecture	Resolution	mAP
Multigrain	ResNet-50	$224^2$	75.1
Multigrain	ResNet-50	largest side 800	82.5
Supervised	ViT-B/16	$224^2$	76.4
DINO	ViT-B/16	$224^2$	81.7
DINO	ViT-B/8	$320^2$	85.5

Video Segmentation and Attention Masks (Table 5, Figures 3–4)

On DAVIS-2017 video object segmentation the authors followed Jabri et al.’s protocol: frozen patch features at 480p, labels propagated by nearest neighbors between consecutive frames, nothing trained for the task. DINO reached a $(\mathcal{J}\&\mathcal{F})_m$ of 61.8 (ViT-S/16), 62.3 (ViT-B/16), 69.9 (ViT-S/8) and 71.4 (ViT-B/8), against 66.0 for a supervised ViT-S/8, 67.6 for the best self-supervised video method listed (STC) and 81.8 for the supervised video model STM. Since nothing is fine-tuned, the authors conclude that the output tokens retain spatial information; the /8 models did much better, by 9.1 points for ViT-B.

Figure 3 shows that different heads of the last layer of ViT-S/8 attend to different objects or parts, at 480p, which gives sequences of 3,601 tokens. Figure 4 turns the [CLS] attention into masks by keeping the patches that hold 60% of the attention mass, takes the best head, and scores the masks by Jaccard similarity (intersection over union) with the ground truth on PASCAL VOC 2012 validation images:

Weights	ViT-S/16	ViT-S/8
Random	22.0	21.8
Supervised	27.3	23.7
DINO	45.9	44.7

The authors stress that the attention maps are smooth and not optimized to produce masks, and that self-supervised convnets also hold segmentation information but need dedicated methods to extract it.

Attention Masks in Code

The mask rule is simple to state and has one free parameter, the mass fraction. The NumPy script below applies it to synthetic attention maps on the $28 \times 28$ patch grid of a ViT with $8 \times 8$ patches at $224 \times 224$, with a known elliptical object. One map follows the object, one adds attention on a background structure, one is nearly uniform. The maps are constructed, not produced by a network.

import numpy as np
from scipy.ndimage import gaussian_filter

# A 224x224 scene seen by a ViT with 8x8 patches: a 28x28 grid of patch tokens.
S, P = 224, 8
G = S // P
yy, xx = np.mgrid[0:S, 0:S]
gt = ((yy - 120) / 60) ** 2 + ((xx - 100) / 45) ** 2 < 1          # ground-truth object (ellipse)
gt_patch = gt.reshape(G, P, G, P).mean(axis=(1, 3)) >= 0.5          # patch is object if half its pixels are

def keep_mass(attn, frac):
    """Smallest set of patches holding `frac` of the attention mass (Caron et al.'s Fig. 4 rule)."""
    a = attn.ravel() / attn.sum()
    order = np.argsort(-a, kind="stable")
    n = np.searchsorted(np.cumsum(a[order]), frac) + 1           # patches needed to reach frac
    mask = np.zeros(a.size, bool)
    mask[order[:n]] = True
    return mask.reshape(attn.shape)

def jaccard(m, g):
    return (m & g).sum() / (m | g).sum()

rng = np.random.default_rng(0)
noise = rng.random((G, G))
blob = gaussian_filter(gt_patch.astype(float), 1.0)                 # smooth map that follows the object
clutter = np.zeros((G, G)); clutter[3:7, 20:26] = 1                 # a background structure
maps = {
    "object-focused": blob + 0.02 * noise,
    "with clutter":   blob + 0.6 * gaussian_filter(clutter, 1.0) + 0.15 * noise,
    "uniform":        np.ones((G, G)) + 0.02 * noise,
}
print(f"{G}x{G} patches, object covers {gt_patch.sum()} of {G * G}")
print(f"{'attention map':16s}" + "  ".join(f"{f'keep {int(f * 100)}%':>9s}" for f in (0.4, 0.6, 0.8, 0.9)))
for name, attn in maps.items():
    row = []
    for f in (0.4, 0.6, 0.8, 0.9):
        m = keep_mass(attn, f)
        row.append(f"{jaccard(m, gt_patch):5.3f}/{m.sum():3d}")
    print(f"{name:16s}" + "  ".join(row))
print("cells: Jaccard with the object / number of patches kept")

Output (NumPy 2.3.5, SciPy 1.16.3; identical on two runs):

28x28 patches, object covers 129 of 784
attention map    keep 40%   keep 60%   keep 80%   keep 90%
object-focused  0.426/ 55  0.651/ 84  0.938/121  0.832/155
with clutter    0.605/ 78  0.940/129  0.489/264  0.304/424
uniform         0.097/312  0.135/469  0.160/626  0.160/705
cells: Jaccard with the object / number of patches kept

The mass rule adapts the mask size to how concentrated the attention is: a focused map needs few patches to reach a given fraction, a diffuse one needs many, so a near-uniform map, the analogue of random weights, keeps about the stated fraction of all patches and scores low at every fraction. The best fraction depends on the map. For the focused map it is near 80%, because the smooth fall-off at the object’s border carries mass; with clutter it is 60%, because beyond that the mask spills into the background structure. A single fixed fraction is therefore a convention that favors some maps over others, which matters for an inconsistency in the paper described under Claims Versus Evidence.

Fine-Tuning Transfer (Table 6)

Fine-tuned with DeiT’s protocol, top-1 accuracy:

Model	Pre-training	CIFAR-10	CIFAR-100	iNat18	iNat19	Flowers	Cars	ImageNet
ViT-S/16	Supervised	99.0	89.5	70.7	76.6	98.2	92.1	79.9
ViT-S/16	DINO	99.0	90.5	72.0	78.2	98.5	93.0	81.5
ViT-B/16	Supervised	99.0	90.8	73.2	77.7	98.4	92.1	81.8
ViT-B/16	DINO	99.1	91.7	72.6	78.6	98.8	93.0	82.8

DINO pre-training raised ImageNet accuracy by 1.6 points for ViT-S/16 and 1.0 for ViT-B/16. In Appendix A (arXiv Table 11; supplement Table 2), a supervised ViT-B/16 initialized with DINO reached 82.8% against 81.8% from random initialization and 81.9% from supervised pre-training on ImageNet, so, the authors argue, the gain is not simply longer training.

Ablations (Table 7, Figure 5, arXiv §5.2–5.5)

Table 7 (ViT-S/16, 300 epochs, k-NN / linear) reads: DINO 72.8 / 76.1; without the momentum teacher 0.1 / 0.1; with Sinkhorn–Knopp added 72.2 / 76.0; without multi-crop 67.9 / 72.5; with a mean squared error loss 52.6 / 62.4; with a predictor 71.8 / 75.6; and the authors’ BYOL, MoCo-v2 and SwAV runs at 66.6 / 71.4, 62.0 / 71.6 and 64.7 / 71.8. Without momentum, only a stronger normalization such as Sinkhorn–Knopp prevented collapse (the SwAV row). Figure 5 plots k-NN accuracy against throughput for patch sizes 16, 8 and 5: accuracy rose as patches shrank, at no parameter cost, while throughput fell to 44 images per second for $5 \times 5$ against 180 for $8 \times 8$.

The analyses that ICCV moved to its supplement report that a teacher copied from the student, or from the previous iteration, did not converge, while one frozen for an epoch reached 66.6% k-NN against 72.8% for the momentum teacher, which outperformed the student throughout training (arXiv Figure 6; supplement Figure 2). Removing either centering or sharpening drove the KL term to zero, with the teacher entropy going to 0 without centering and to $-\log(1/K)$ without sharpening (arXiv Figure 7; supplement Figure 3). The DINO article explains these results. The compute study, the source of the “two servers, three days” claim, used ViT-S/16 on two 8-GPU machines (arXiv Table 8; supplement Table 7), with top-1 linear accuracy, total training time and peak memory per GPU:

Crops	100 epochs: top-1	time	300 epochs: top-1	time	memory
$2 \times 224^2$	67.8	15.3 h	72.5	45.9 h	9.3 GB
$2 \times 224^2 + 2 \times 96^2$	71.5	17.0 h	74.5	51.0 h	10.5 GB
$2 \times 224^2 + 6 \times 96^2$	73.8	20.3 h	75.9	60.9 h	12.9 GB
$2 \times 224^2 + 10 \times 96^2$	74.6	24.2 h	76.1	72.6 h	15.4 GB

With ten local crops, 100 epochs gave 74.6% in about 24 hours against 72.5% for 300 two-crop epochs in about 46, which the authors summarize as two points more in half the time, at higher memory. The 76.1% in 72.6 hours is the three-day figure; it is below the 77.0% of Table 2, which comes from 800 epochs and the four-layer linear probe. With k-NN accuracy after 100 epochs without multi-crop, batch sizes of 128, 256, 512 and 1,024 gave 57.9%, 59.1%, 59.6% and 59.9%, the 128 run on a single GPU, and a batch of 8 reached 35.2% after 50 epochs (arXiv Table 9; supplement Table 8).

Among the appendix ablations, adding batch normalization to the head changed 100-epoch k-NN accuracy from 69.7% to 68.6% (Appendix C), and raising ViT-S’s heads from 6 to 16 at constant width raised k-NN accuracy from 72.8% to 73.8% while throughput fell from 1,007 to 860 images per second; all main experiments kept DeiT’s 6 heads (Appendix D). Multi-crop helped frameworks unequally (Appendix E): six extra $96^2$ crops raised linear accuracy by 3.4 points for DINO, 3.3 for SwAV and 1.8 for MoCo-v2, and lowered it by 6.6 for BYOL, whose k-NN accuracy with multi-crop rose faster early in training and then declined; sweeps of learning rate, weight decay and crop number showed the same pattern, except at low learning rates, where accuracy kept rising but stayed low.

Claims Versus Evidence

Several headline statements are qualified by the paper’s own tables.

• Segmentation is a property of self-supervised ViTs, not of DINO. The abstract says segmentation information does not emerge as clearly with supervised ViTs or convnets. Figure 4 supports the comparison with supervised ViTs. Convnets were not measured with the same protocol; the paper only notes that their segmentation information needs dedicated extraction methods. And §1 concedes that the emergence of masks “seems to be a property shared across self-supervised methods”. Appendix D confirms it: ViT-S/16 masks scored 46.3 with MoCo-v2, 47.8 with BYOL and 46.8 with SwAV, all above DINO’s 45.9. What the tables tie to DINO specifically is the k-NN property.
• 60% or 80% of the mass. That Appendix D table describes its masks as keeping 80% of the attention mass, while Figure 4 describes masks keeping 60%, and both report the same random, supervised and DINO values (22.0, 27.3, 45.9). One of the two descriptions is wrong in every version read; the paper does not say which.
• “Almost on par”. The k-NN gap is 2.5 points for ViT-S/16 and 1.4 for ViT-S/8, small next to the 4.8 to 8.3 points of the other methods on ViT-S, but not zero.
• Transfer. The caption of Table 6 says DINO transfers better than supervised pre-training. It does on most entries, ties on CIFAR-10 for ViT-S/16, and is lower on iNat18 for ViT-B/16 (72.6 against 73.2).
• Baselines. The ViT-S baselines were the authors’ own runs, BYOL at 300 epochs against 800 for the others, and Appendix B says their ResNet-50 reproductions matched or beat the published numbers. DINO’s ViT linear results use enlarged probe inputs, four concatenated [CLS] tokens for ViT-S and [CLS] plus pooled patch tokens for ViT-B, which add 0.9 points for ViT-S/16 and 0.2 for ViT-B/16 (Appendix F).
• Cross-references. The ICCV supplement’s Appendix D says the teacher also beats the student with a ResNet-50 “(Appendix D)”, but the curve showing it appears only in the arXiv appendix. Smaller labeling slips survive the DeiT-S to ViT-S renaming: Figure 5’s legend, the caption of arXiv Table 13 (supplement Table 4), and the caption of supplement Table 7 and the compute paragraph around it still say DeiT-S.

Impact

The arXiv conclusion states that “the main result” is evidence that self-supervised learning could be the key to a BERT-like model based on ViT, and announces plans to pre-train a large ViT with DINO on random uncurated images. Both sentences, and a pointer to ViT image-retrieval work, are absent from the ICCV conclusion, which keeps only the two properties and their uses for retrieval and weakly supervised segmentation. The paper is widely cited, and its influence ran along both properties.

Dense and unsupervised localization. LOST (Siméoni et al., 2021) localized objects in single images from the keys of the last attention layer of a DINO ViT-S/16, without labels, and outperformed earlier object discovery methods by up to 8 CorLoc points on PASCAL VOC 2012. STEGO (Hamilton et al., 2022) started from the observation that correlations between the dense features of unsupervised networks such as DINO are already semantically consistent and distilled them into unsupervised semantic segmentation, improving the prior state of the art by 14 mIoU on CocoStuff and 9 on Cityscapes.

The self-distillation line. iBOT (Zhou et al., 2022) took DINO’s loss on the [CLS] token, which it describes as self-distillation “proposed recently in DINO”, and added a masked-patch term with the teacher as an online tokenizer. DINOv2 (Oquab et al., 2024) combined the DINO and iBOT losses and scaled data and models; re-evaluating the released DINO ViT-S/8 in its Table 4, it measured 78.6% k-NN and 79.2% linear on ImageNet, against 78.3% and 79.7% in Caron et al.’s Table 2. The DINO article follows this line through DINOv3.

Clean attention as an exception. Darcet et al. (2024) found high-norm artifact tokens in the attention maps of supervised DeiT-III, text-supervised OpenCLIP and self-supervised DINOv2, but not in the original DINO, and concluded that DINO is the exception rather than the rule among ViTs. In the DINOv3 report, DINO’s ViT-S/16 and ViT-B/16 still scored 61.1 and 60.1 CorLoc on VOC 2007 for TokenCut object discovery on output patch features, above DINOv2 ViT-g/14 (55.6) and below only DINOv3’s 7B model (66.1) among the backbones compared (Siméoni et al., 2025, Figure 14).

Limitations

The authors name several. Training used ImageNet only, a curated dataset, and the arXiv conclusion leaves uncurated data to future work. Collapse prevention is a balance: the paper’s own ablations collapse without the momentum teacher, with a centering rate of 0.999, and with a fixed teacher temperature above 0.06. The best features need small patches, which cost throughput and memory. Small batches fell slightly short and, the authors expect, would need re-tuned momentum rates; multi-crop did not combine with BYOL, and the reasons for the k-NN property were left open, with the concurrent CsMI named as a lead.

Later work adds to these. The DINOv3 report notes that the best object discovery results with DINO used a hand-picked feature, the keys of the last attention layer, a choice that does not consistently carry over to other backbones (Siméoni et al., 2025, §6.1.4). Linear-probe numbers depend on the protocol: DINOv2’s re-evaluation of DINO ViT-S/8 came out half a point below the paper’s. And absolute accuracies were soon overtaken: DINOv2’s frozen ViT-g/14 reached 86.5% linear on ImageNet in the same table.

What Came After

The method’s lineage, iBOT’s masked self-distillation, DINOv2’s curated LVD-142M data and billion-parameter teacher, and DINOv3’s Gram anchoring, is described in the DINO article. Masked image modeling, as in the masked autoencoder, developed in parallel as the reconstruction-based alternative, and representation learning places both among the ways of learning features. The question the paper posed, what self-supervision gives a ViT that labels do not, became central to work on frozen general-purpose backbones.