DINO

DINO is a self-supervised method, and the family of models built on it, in which a student network is trained to reproduce the output of a teacher network on a different crop of the same image, with no labels anywhere. The teacher is not a separate pretrained model: it is an exponential moving average of the student’s own weights. Mathilde Caron and colleagues at Facebook AI Research and Inria introduced it in “Emerging Properties in Self-Supervised Vision Transformers”, posted to arXiv in April 2021 and presented at ICCV 2021, and described it as a form of self-distillation with no labels, the phrase the name is drawn from, although the paper never spells the acronym out (Caron et al., 2021). Trained on Vision Transformers, DINO produced two properties that drew attention: the last layer’s class-token attention outlined objects without any segmentation supervision, and the features worked as a k-nearest-neighbor classifier almost as well as a trained linear one. DINOv2 (Oquab et al., 2024) and DINOv3 (Siméoni et al., 2025) kept the self-distillation core, added a masked-patch objective, curated data at scale and grew the teacher to 1.1 billion and then 6.7 billion parameters, aiming at a backbone used frozen for many tasks.

DINO belongs to the non-contrastive branch of self-supervised learning, which surveys the field and its other families; representation learning places it among the other ways of learning features. Two unrelated models share the name. DINO for object detection (Zhang et al., 2023), “DETR with Improved DeNoising Anchor Boxes”, is a supervised DETR variant, and Grounding DINO is an open-set detector built on that detector; neither uses self-distillation.

Problem Addressed

By 2020, self-supervised methods on convolutional networks had closed much of the gap to supervised ImageNet pre-training, using negatives (contrastive learning), clustering, or a predictor with a momentum target (BYOL; Grill et al., 2020). Vision Transformers were competitive with convnets but, in the authors’ framing, had not shown clear benefits: they needed more data and compute, and their features showed no distinctive properties. Caron et al. asked whether supervision itself was the limit, since language transformers had succeeded with self-supervised pre-training and a single image label discards most of what an image contains (Caron et al., 2021, §1).

They studied which ingredients of existing methods matter for ViTs and kept a small set: a momentum encoder, multi-crop augmentation and a cross-entropy loss between softmax outputs. The resulting method needs no negatives, no predictor and no batch normalization in the head, and works on both ViTs and ResNets without changes. Its one anti-collapse device is to center and sharpen the teacher’s output.

Architecture

Student and teacher share one architecture $g = h \circ f$, a backbone $f$ followed by a projection head $h$, with different weights $\theta_s$ and $\theta_t$. Only $f$ is kept after training; the head exists for the loss (Caron et al., 2021, §3.1).

Backbone

The paper uses DeiT-style ViTs, ViT-S (21M parameters) and ViT-B (85M), with $16 \times 16$ or $8 \times 8$ patches, written ViT-S/16, ViT-B/8 and so on, and a ResNet-50 for comparison. The projection head is attached to the [CLS] token, which the authors keep calling the class token although no class label is involved. Smaller patches add no parameters but lengthen the token sequence: ViT-S/8 has 785 tokens at $224 \times 224$ against 197 for ViT-S/16, and ran at 180 images per second on a V100 against 1,007 (Table 1). In the paper’s patch-size study, smaller patches raised k-NN accuracy at that cost in throughput (Figure 5).

Projection Head

The head is a three-layer MLP with hidden width 2,048, a 256-dimensional $\ell_2$-normalized bottleneck, and a weight-normalized linear layer to $K$ outputs, similar to SwAV’s design (Caron et al., 2020). The default is $K = 65{,}536$; in a 100-epoch ablation $K$ from 4,096 to 262,144 gave k-NN accuracies within 0.6 points of each other, while 1,024 was 1.5 points lower than 4,096 (arXiv v2, Appendix C). There is no predictor, so student and teacher are architecturally identical, and with ViTs the whole system has no batch normalization.

Multi-Crop Views

Each image yields a set $V$ of views: two global crops resized to $224 \times 224$ and covering a large part of the image, and several local crops at $96 \times 96$ covering small parts. Global crops cover a fraction $(s, 1)$ of the image area and local crops $(0.05, s)$, ranges the authors made non-overlapping following SwAV; their ablation found $s$ around 0.3 best (Appendix E). All crops go through the student; only the global ones go through the teacher, so the student must predict, from a small patch of the scene, what the teacher says about the whole: the paper calls this “local-to-global” correspondence. Augmentations follow BYOL: color jitter, Gaussian blur and solarization.

Training

Self-Distillation Objective

In knowledge distillation a small student learns to match the softened outputs of a fixed, larger teacher (Hinton et al., 2015). DINO keeps the loss and drops the fixed teacher. The student’s output is turned into a distribution over $K$ dimensions with a temperature $\tau_s$:

$$P_s(x)^{(i)} = \frac{\exp\big(g_{\theta_s}(x)^{(i)} / \tau_s\big)}{\sum_{k=1}^{K} \exp\big(g_{\theta_s}(x)^{(k)} / \tau_s\big)},$$

and the teacher’s similarly, after subtracting a center $c$ and with its own temperature $\tau_t$:

$$P_t(x) = \mathrm{softmax}\!\left(\frac{g_{\theta_t}(x) - c}{\tau_t}\right).$$

The student minimizes the cross-entropy $H(a, b) = -\sum_i a^{(i)} \log b^{(i)}$ between the teacher’s distribution on each global view and its own on every other view:

$$\min_{\theta_s} \sum_{x \in \{x^g_1, x^g_2\}} \; \sum_{\substack{x' \in V \\ x' \neq x}} H\big(P_t(x), P_s(x')\big).$$

Gradients flow only into the student (a stop-gradient on the teacher). DINOv2 calls the $K$ outputs “prototype scores”; no clustering step is run.

Momentum Teacher

After each step the teacher’s weights move toward the student’s:

$$\theta_t \leftarrow \lambda\, \theta_t + (1 - \lambda)\, \theta_s,$$

with $\lambda$ following a cosine schedule from 0.996 to 1. This is the momentum encoder of MoCo (He et al., 2020), but here there is no queue and no contrastive loss; the authors compare it instead to the mean teacher of semi-supervised learning and to Polyak–Ruppert averaging, a running ensemble of past students. In their training curves the teacher had higher k-NN accuracy than the student throughout training, so it supplies better targets than the student could. Copying the student into the teacher, or using the student from the previous iteration, did not converge; using the student from the previous epoch did not collapse and reached 66.6% k-NN accuracy, against 72.8% for the momentum teacher (Caron et al., 2021, arXiv v2 §5.2, Figure 6). Sections 5.2–5.5 of the arXiv version, with Figures 6–7 and Tables 8–9, sit in the supplementary material of the ICCV version (Appendix D, as Figures 2–3 and Tables 7–8); this article uses the arXiv numbering.

Centering and Sharpening

A network trained to agree with itself can collapse: give every image the same output and the loss is minimal. DINO distinguishes two forms. Either one output dimension dominates for every input, or the output is uniform. The two teacher operations push in opposite directions (§3.1; arXiv v2 §5.3):

• Centering subtracts $c$, a running mean of the teacher’s outputs over batches of size $B$, updated as $c \leftarrow m\, c + (1 - m) \frac{1}{B} \sum_{i=1}^{B} g_{\theta_t}(x_i)$. A dimension that is high for every image is pulled down, which prevents domination but encourages a uniform output.
• Sharpening uses a low teacher temperature, $\tau_t$ warmed from 0.04 to 0.07 over the first 30 epochs against $\tau_s = 0.1$, which makes the targets peaked and pushes away from uniform.

The cross-entropy splits as

$$H(P_t, P_s) = h(P_t) + D_{\mathrm{KL}}(P_t \,\|\, P_s),$$

where $h$ is the teacher’s entropy. A KL term at zero means student and teacher agree whatever the input, the signature of collapse. With either operation removed the KL term went to zero, while the entropy went to 0 without centering and to $\log K$ without sharpening (arXiv v2 Figure 7). Centering uses only first-order batch statistics, so the method depends little on batch size: 100-epoch k-NN accuracy ranged from 57.9% at batch 128 to 59.9% at 1,024 (arXiv v2 Table 9). Its smoothing rate was robust from $m = 0$ to $0.99$ and collapsed at $0.999$; a fixed $\tau_t$ above 0.06 collapsed unless it was reached by warm-up (Appendix D).

Recipe and Ablations

ViT-S/16 was trained with AdamW at batch size 1,024 on 16 GPUs, a learning rate of $0.0005 \times \text{batch size}/256$ after a 10-epoch warm-up, and weight decay rising from 0.04 to 0.4. In the ViT-S/16 component ablation at 300 epochs (Table 7), the default reached 72.8% k-NN and 76.1% linear accuracy on ImageNet; without the momentum teacher it collapsed to 0.1%; without multi-crop it reached 67.9% k-NN; with a mean squared error loss in place of cross-entropy, 52.6%; adding a BYOL-style predictor changed little (71.8%). Multi-crop also paid for itself: with two global and ten local crops, 100 epochs took 24.2 hours on two 8-GPU machines and gave 74.6% linear accuracy, against 72.5% for 300 epochs with two crops only, which took 45.9 hours (arXiv v2 Table 8).

A Tiny Self-Distillation Run

The PyTorch script below (version 2.14.1, one CPU thread, about 24 seconds, identical output on two runs of each seed) trains the DINO objective on $48 \times 48$ synthetic images of squares, discs, triangles and crosses with random size, position and brightness. Each step draws 32 images and makes two global views ($24 \times 24$, 40–100% of the area) and two local views ($12 \times 12$, 10–40%), with flips, contrast and brightness jitter and noise. The backbone is a three-layer CNN with batch normalization, as in a ResNet, and global average pooling to 32 dimensions; the head has a 32-dimensional normalized bottleneck and $K = 256$ prototypes. The teacher is an EMA with constant $\lambda = 0.99$, shorter-sighted than DINO’s 0.996, to suit a 600-step run. The same seed trains three times: with centering and sharpening ($\tau_t = 0.04$), with sharpening only, and with centering only ($\tau_t = \tau_s = 0.1$). The shape labels are used only for the k-NN evaluation, a weighted 20-NN vote among 200 labelled images, scored on 400 others.

import math
import sys
import numpy as np
import torch
from torch import nn
import torch.nn.functional as F

torch.set_num_threads(1)                      # one thread: identical floating-point sums on every run
print(f"torch {torch.__version__}")

def render(n, rng, size=48):
    """n grayscale 48x48 images of four shapes (square, disc, triangle, cross) with random
    size, position, shape brightness and background brightness."""
    yy, xx = np.mgrid[0:size, 0:size].astype(np.float32)
    X = np.empty((n, 1, size, size), np.float32)
    y = rng.integers(0, 4, n)
    for i in range(n):
        r = rng.uniform(8, 14)
        cy, cx = rng.uniform(r + 2, size - r - 2, 2)
        dy, dx = yy - cy, xx - cx
        if y[i] == 0:   m = (abs(dy) < r) & (abs(dx) < r)
        elif y[i] == 1: m = dy ** 2 + dx ** 2 < r ** 2
        elif y[i] == 2: m = (abs(dy) < r) & (abs(dx) < (dy + r) / 2)
        else:           m = ((abs(dy) < r) & (abs(dx) < r / 3)) | ((abs(dx) < r) & (abs(dy) < r / 3))
        X[i, 0] = rng.uniform(0.1, 0.4) + m * rng.uniform(0.3, 0.6)
    return torch.from_numpy(X), torch.from_numpy(y)

def crops(x, out, smin, smax, g):
    """Random resized crops covering a fraction [smin, smax] of the image area, resampled to
    out x out, with random flips, contrast and brightness jitter and pixel noise."""
    B = x.shape[0]
    s = torch.sqrt(smin + (smax - smin) * torch.rand(B, generator=g))   # side fraction
    theta = torch.zeros(B, 2, 3)
    theta[:, 0, 0] = s * torch.where(torch.rand(B, generator=g) < 0.5, -1.0, 1.0)
    theta[:, 1, 1] = s
    theta[:, :, 2] = (1 - s)[:, None] * (2 * torch.rand(B, 2, generator=g) - 1)
    grid = F.affine_grid(theta, (B, 1, out, out), align_corners=False)
    v = F.grid_sample(x, grid, align_corners=False)
    c = 0.6 + 0.8 * torch.rand(B, 1, 1, 1, generator=g)
    b = 0.2 * torch.rand(B, 1, 1, 1, generator=g) - 0.1
    return c * (v - 0.4) + 0.4 + b + 0.05 * torch.randn(v.shape, generator=g)

class Backbone(nn.Module):                     # small CNN, global average pooling -> 32-d feature
    def __init__(self):
        super().__init__()
        layers, c = [], 1
        for o in (8, 16, 32):
            layers += [nn.Conv2d(c, o, 3, padding=1), nn.BatchNorm2d(o), nn.GELU()]
            layers += [nn.MaxPool2d(2)] if o < 32 else []
            c = o
        self.net = nn.Sequential(*layers)
    def forward(self, x):
        return self.net(x).mean((2, 3))

class Head(nn.Module):                         # DINO-style head: MLP, l2 bottleneck, K prototypes
    def __init__(self, K):
        super().__init__()
        self.mlp = nn.Sequential(nn.Linear(32, 64), nn.GELU(), nn.Linear(64, 32))
        self.W = nn.Parameter(torch.randn(K, 32))
    def forward(self, f):                      # cosine between the bottleneck and each prototype
        return F.normalize(self.mlp(f), dim=1) @ F.normalize(self.W, dim=1).T

def knn_accuracy(backbone, Xtr, ytr, Xte, yte, k=20, T=0.07):
    """Weighted 20-NN vote on cosine similarity of frozen features, the weighting DINO uses."""
    backbone.eval()
    with torch.no_grad():
        f = lambda X: F.normalize(backbone(F.interpolate(X, size=24, mode="bilinear", antialias=True)), dim=1)
        sim, idx = (f(Xte) @ f(Xtr).T).topk(k, dim=1)
    backbone.train()
    votes = torch.zeros(len(Xte), 4).scatter_add_(1, ytr[idx], (sim / T).exp())
    return (votes.argmax(1) == yte).float().mean().item()

def entropy(p):
    return -(p * p.clamp_min(1e-12).log()).sum(-1)

def train(center, tau_t, seed, steps=600, B=32, K=256, tau_s=0.1, lam=0.99, m=0.9):
    torch.manual_seed(seed)
    g = torch.Generator().manual_seed(seed)
    rng = np.random.default_rng(seed)
    X, _ = render(2000, rng)                   # unlabeled images for self-distillation
    Xtr, ytr = render(200, rng)                # labelled k-NN memory (about 50 per shape)
    Xte, yte = render(400, rng)                # labelled test images
    student = nn.Sequential(Backbone(), Head(K))
    teacher = nn.Sequential(Backbone(), Head(K))
    teacher.load_state_dict(student.state_dict())
    for p in teacher.parameters():
        p.requires_grad_(False)
    knn_init = knn_accuracy(student[0], Xtr, ytr, Xte, yte)
    opt = torch.optim.AdamW(student.parameters(), lr=1e-3, weight_decay=0.04)
    C = torch.zeros(K)                         # center of the teacher's logits
    for step in range(steps):
        x = X[torch.randint(0, len(X), (B,), generator=g)]
        glob = [crops(x, 24, 0.4, 1.0, g) for _ in range(2)]    # two global views
        loc = [crops(x, 12, 0.1, 0.4, g) for _ in range(2)]     # two local views
        with torch.no_grad():
            t_out = [teacher(v) for v in glob]                  # teacher sees global views only
            t_prob = [F.softmax((t - C) / tau_t, dim=1) for t in t_out]
        s_logp = [F.log_softmax(student(v) / tau_s, dim=1) for v in glob + loc]
        pairs = [(i, j) for i in range(2) for j in range(len(s_logp)) if j != i]
        loss = sum(-(t_prob[i] * s_logp[j]).sum(1).mean() for i, j in pairs) / len(pairs)
        opt.zero_grad()
        loss.backward()
        opt.step()
        with torch.no_grad():
            for pt, ps in zip(teacher.parameters(), student.parameters()):
                pt.mul_(lam).add_((1 - lam) * ps)               # EMA (momentum) teacher
            if center:
                C = m * C + (1 - m) * torch.cat(t_out).mean(0)
    with torch.no_grad():                      # collapse diagnostics on 256 images, two views each
        x = X[:256]
        pt = F.softmax((teacher(crops(x, 24, 0.4, 1.0, g)) - C) / tau_t, dim=1)
        ps = F.softmax(student(crops(x, 24, 0.4, 1.0, g)) / tau_s, dim=1)
        h = entropy(pt).mean().item()                           # per-image teacher entropy
        h_mean = entropy(pt.mean(0)).item()                     # entropy of the batch average
        kl = (pt * (pt.clamp_min(1e-12).log() - ps.clamp_min(1e-12).log())).sum(1).mean().item()
        used = len(torch.unique(pt.argmax(1)))                  # prototypes that win an image
    return h, h_mean, kl, used, knn_init, knn_accuracy(teacher[0], Xtr, ytr, Xte, yte)

seed = int(sys.argv[1]) if len(sys.argv) > 1 else 0
print(f"seed {seed}, K = 256, log K = {math.log(256):.2f}")
print(f"{'setting':24s} {'H(teacher)':>10s} {'H(mean)':>8s} {'KL':>6s} {'winners':>7s} {'k-NN before':>11s} {'after':>6s}")
for name, center, tau_t in [("centering + sharpening", True, 0.04),
                            ("sharpening only", False, 0.04),
                            ("centering only", True, 0.1)]:
    h, h_mean, kl, used, k0, k1 = train(center, tau_t, seed)
    print(f"{name:24s} {h:10.2f} {h_mean:8.2f} {kl:6.3f} {used:7d} {k0:11.3f} {k1:6.3f}")

Output of python tiny_dino.py 0 and python tiny_dino.py 1:

torch 2.14.1
seed 0, K = 256, log K = 5.55
setting                  H(teacher)  H(mean)     KL winners k-NN before  after
centering + sharpening         3.43     4.49  0.508      14       0.385  0.832
sharpening only                0.00     0.00  0.009       1       0.385  0.623
centering only                 5.53     5.54  0.007      25       0.385  0.440

torch 2.14.1
seed 1, K = 256, log K = 5.55
setting                  H(teacher)  H(mean)     KL winners k-NN before  after
centering + sharpening         4.01     4.85  0.514      17       0.377  0.678
sharpening only                0.00     0.00  0.009       1       0.377  0.562
centering only                 5.50     5.54  0.019      10       0.377  0.417

The two failure modes appear as the paper describes them. With sharpening alone, the teacher’s per-image entropy and the entropy of its batch-averaged output are both zero and a single prototype wins every image: the dominant-dimension collapse. With centering alone, both entropies sit near $\log K = 5.55$: every image gets a nearly uniform distribution, and which prototype is largest is noise. In both cases the KL term is close to zero, so the loss no longer distinguishes one image from another and the backbone stops learning. With both operations the per-image entropy is well below $\log K$ while the batch average stays broad, meaning that different images are confidently assigned to different groups of prototypes, and the KL term stays near 0.5.

The k-NN column shows what that buys. Only the run with both operations made the frozen backbone a markedly better shape classifier than its random initialization, by 45 points on seed 0 and 30 on seed 1; the two-seed spread is wide, in our reading a consequence of 600 steps on a tiny network. The sharpening-only runs also improved, by about 19 and 24 points, which we read as learning that happened before the collapse took hold; the centering-only runs gained only 4 to 6 points.

Inference

At inference only a backbone is kept, without the projection head. An image gives a global descriptor, the [CLS] output, and one feature per patch, and the features are used frozen in four main ways:

• k-NN classification and retrieval. DINO’s k-NN evaluation follows Wu et al. (2018): store the normalized training features, find the 20 nearest by cosine similarity, and let each vote with weight $\exp(\text{sim}/0.07)$. It needs no training or augmentation and one pass over the data (Caron et al., 2021, Appendix F). The same nearest-neighbor search serves image retrieval and copy detection.
• Linear probes. A linear classifier on the frozen [CLS] feature, the standard protocol for comparing self-supervised features; DINOv2’s protocol also tries concatenating average-pooled patch tokens and the outputs of several layers (Oquab et al., 2024, Appendix B.3).
• Dense probes and decoders. A linear layer on the patch tokens gives a low-resolution map for semantic segmentation or monocular depth estimation; heavier decoders such as DPT or Mask2Former can be trained on top while the backbone stays frozen.
• Attention and patch similarity. Thresholding the self-attention of the [CLS] token in the last layer, keeping the patches that hold 60% of its mass, yields a foreground mask; DINOv2 and DINOv3 show patch features through principal component analysis, whose first component separates object from background.

Fine-tuning remains possible and usually adds accuracy (see transfer learning), but the later models were designed so that it is optional.

Performance

All numbers are top-1 accuracy on the ImageNet validation set unless stated otherwise, as reported by the authors.

DINO (2021). Among methods sharing an architecture, DINO tied SwAV’s 75.3% linear accuracy on ResNet-50 and had the best ResNet-50 k-NN accuracy (67.5%), and with ViT-S it reached 77.0% linear and 74.5% k-NN, beating BYOL, MoCo v2 and SwAV by 3.5 points linear and 7.9 points k-NN. The near parity of k-NN and linear accuracy appeared only with DINO and ViTs. The best models were ViT-B/8 at 80.1% linear and ViT-S/8 at 78.3% k-NN (Table 2). Other frozen-feature results: on DAVIS-2017 video object segmentation, propagating labels by nearest neighbors between frames, ViT-B/8 reached 71.4 $(\mathcal{J}\&\mathcal{F})_m$ (Table 5); masks from thresholded attention on PASCAL VOC 2012 validation images reached a Jaccard similarity of 45.9 for ViT-S/16, against 27.3 for a supervised ViT-S/16 (Figure 4). Fine-tuned on ImageNet, DINO pre-training gave 81.5% for ViT-S/16 against 79.9% for supervised pre-training (Table 6).

DINOv2 (2023, TMLR 2024). Frozen ViT-g/14 features reached 86.5% linear and 83.5% k-NN at $224 \times 224$; in the same table, OpenCLIP ViT-G/14 reached 86.2% linear and the original DINO ViT-S/8 79.2%, against 79.7% in Caron et al.’s own Table 2 (Oquab et al., 2024, Table 4). Fine-tuning the ViT-g/14 added 2.0 points (88.5%; Table 5). With linear heads on frozen patch features, ADE20K semantic segmentation reached 49.0 mIoU (53.0 with the boosted multi-scale protocol), against 39.3 for OpenCLIP (Table 10); a frozen ViT-g/14 inside a ViT-Adapter and Mask2Former pipeline reached 60.2 mIoU.

DINOv3 (2025). The 7B model’s frozen dense features led the paper’s comparison: linear ADE20K segmentation 55.9 mIoU at 1,024 patch tokens, against 49.5 for DINOv2 ViT-g/14 and 53.0 for AM-RADIOv2.5 (Siméoni et al., 2025, Table 3). The linear ImageNet probe reached 88.4%, 0.7–0.9 points below the weakly supervised (image–text) SigLIP 2 and Perception Encoder (Table 7). With the backbone frozen and only a Plain-DETR detector trained, the system reached 66.1 mAP on COCO val2017 with test-time augmentation (Table 10). With a frozen backbone, ViT-Adapter and Mask2Former it reached 63.0 mIoU on ADE20K with test-time augmentation; the introduction describes the segmentation result as outperforming specialized fine-tuned pipelines, but Table 11 shows it equal to ONE-PEACE’s 63.0.

Strengths

• Frozen features that transfer. A single forward pass can serve classification, retrieval, segmentation, depth and tracking, which is the use the DINOv2 and DINOv3 papers designed for.
• Good nearest-neighbor geometry. k-NN accuracy close to linear accuracy means the feature space is organized by semantic similarity, which helps retrieval and few-label tasks.
• Spatial information. Without any dense supervision, patch features and attention separate objects and parts; DINOv3’s Gram anchoring targets this property.
• Simple objective. One cross-entropy loss, no negatives and no predictor; centering depends only on running means, so small batches work (arXiv v2 Table 9).
• Labels not needed in any domain. The same recipe trained a satellite model on 493 million unlabeled satellite images (Siméoni et al., 2025, §8).

Limitations

• Sensitive balance. Collapse is prevented only by the balance between centering and sharpening with a slow teacher; the paper’s ablations show collapse at $m = 0.999$, at fixed $\tau_t$ above 0.06 and without the momentum teacher.
• Compute at scale. DINOv2’s ViT-g/14 took an estimated 22,016 A100 GPU-hours to reproduce (Oquab et al., 2024, Table 14); DINOv3 trains a 6.7B-parameter teacher on 256 GPUs and distills it into smaller models for deployment.
• Dense features degrade with long training. DINOv3 reports that segmentation accuracy of the ViT-g and ViT-7B peaked around 200k iterations and then declined while ImageNet accuracy kept rising, the problem Gram anchoring was introduced to fix (Siméoni et al., 2025, §4.1). Large ViTs also produce high-norm artifact tokens in background areas, which Darcet et al. (2024) traced and removed with extra register tokens.
• Curated data blurs “unsupervised”. LVD-142M was assembled by retrieving web images similar to the training sets of many benchmarks, including ADE20K, Cityscapes and fine-grained classification datasets, with near-duplicates of test images removed (Oquab et al., 2024, §3 and Table 15). In our reading, part of DINOv2’s transfer comes from data chosen to resemble the evaluation domains.
• Bias. On Dollar Street, DINOv2’s ViT-g/14 scored 74.0% in Africa against 89.7% in Europe, and 67.4% for low-income against 90.5% for high-income households (Oquab et al., 2024, Table 12). The text of §8.1 gives the gaps as 25.7% and 31.7%, which do not match the table’s differences of 15.7 and 23.1 points.
• No language. The features carry no text alignment. DINOv3 adds it after training by learning a text encoder against the frozen image encoder, with two added transformer layers on the image side (Siméoni et al., 2025, §5.3); the DINOv2 repository provides the same dino.txt approach.

Historical Importance

DINO showed in 2021 that a self-supervised Vision Transformer has properties a supervised one lacks: attention maps that segment objects and features usable by a nearest-neighbor classifier. Its self-distillation loss became the image-level term of iBOT, and iBOT, DINOv2 and DINOv3 all kept the momentum teacher and multi-crop.

DINOv2 changed the goal from beating other methods on ImageNet to producing general-purpose frozen features, and the DINOv3 authors describe it as the first self-supervised model to match or surpass open-source CLIP variants on a range of tasks (Siméoni et al., 2025, §2). The DINOv3 paper cites histopathology, biology, medical imaging, remote sensing and astronomy among the domains shown to benefit from foundation models like DINOv2 (§1).

Variants

DINO (2021)

The original method, described above: ViT-S, ViT-B and ResNet-50 models trained on ImageNet-1k without labels, plus a ViT-S/16 trained on the 1.2 million clean images of Google Landmarks v2 for retrieval (Caron et al., 2021, Table 3).

iBOT (2021)

iBOT added a patch-level term to DINO’s image-level one: the student receives a view with some patches masked and predicts, for each masked patch, the teacher’s softmax output on the visible patch, so the teacher acts as an online tokenizer for masked image modeling (Zhou et al., 2022). It is the direct parent of DINOv2’s objective; the masked autoencoder is the reconstruction-based alternative.

DINOv2 (2023)

Oquab et al. (2024) describe their objective as a combination of the DINO and iBOT losses, $\mathcal{L}_{\mathrm{DINO}} = -\sum p_t \log p_s$ on class tokens and $\mathcal{L}_{\mathrm{iBOT}} = -\sum_i p_{t,i} \log p_{s,i}$ over masked patch indices $i$, with separate heads for the two terms. They replace the teacher’s centering with the Sinkhorn–Knopp batch normalization of SwAV (three iterations), and add the KoLeo regularizer, derived from the Kozachenko–Leonenko entropy estimator, which spreads $\ell_2$-normalized features within a batch:

$$\mathcal{L}_{\mathrm{koleo}} = -\frac{1}{n} \sum_{i=1}^{n} \log d_{n,i}, \qquad d_{n,i} = \min_{j \neq i} \lVert x_i - x_j \rVert.$$

In their ablations KoLeo raised Oxford-M retrieval by more than 8 points and the masked-image term raised ADE20K linear segmentation by almost 3 mIoU (Table 3). Other changes: patch size 14, 128k prototypes, teacher momentum from 0.994, a final short phase at $518 \times 518$, and engineering for scale (their own FlashAttention implementation, sequence packing, an efficient stochastic depth and fully sharded data parallel training). The largest model, ViT-g/14 (1.1B parameters, 40 blocks, width 1,536), was trained from scratch; the released ViT-S, ViT-B and ViT-L were distilled from it while frozen, and the distilled ViT-L beat a ViT-L trained from scratch on all 12 benchmarks (§6.5).

The data are as much a contribution as the loss. LVD-142M starts from 1.2 billion unique web images, left after unsafe-content filtering, hash deduplication and blurring of identifiable faces; a copy-detection pipeline then removed near-duplicates, including those of benchmark test and validation images. Images are then retrieved for being close, in the embedding of a self-supervised ViT-H/16, to images of curated datasets: ImageNet-22k, ImageNet-1k’s training split, Google Landmarks and many fine-grained, segmentation, depth and retrieval sets. The result is 142,109,386 images, built without labels, text or metadata (§3, Table 15). In a matched ablation, LVD-142M beat 142 million uncurated images on six of seven benchmarks and ImageNet-22k on all but ImageNet-1k (Table 2). In October 2023 the repository added backbones with registers, extra tokens that absorb the high-norm artifacts (Darcet et al., 2024).

DINOv3 (2025)

Siméoni et al. (2025) scale further and target dense features.

Data. From about 17 billion Instagram images that had passed platform moderation, they combine LVD-1689M (1,689 million images selected by hierarchical k-means with balanced sampling over DINOv2 embeddings), a retrieval-curated part, and raw ImageNet-1k, ImageNet-22k and Mapillary Street-level Sequences; 10% of batches are ImageNet-1k alone (§3.1).

Model. A custom ViT-7B (6.7B parameters, 40 blocks, width 4,096, patch 16, 4 registers) with rotary position embeddings whose coordinate box is randomly rescaled during training, 256k DINO prototypes and 96k iBOT prototypes (Table 2).

Objective and schedule. The DINO and iBOT losses with Sinkhorn–Knopp, plus KoLeo with weight 0.1, trained with constant learning rate, weight decay and teacher momentum for 1M iterations at batch size 4,096, with 2 global ($256 \times 256$) and 8 local ($112 \times 112$) crops (§3.2).

Gram anchoring. Patch features become less local as training continues, so a refinement phase adds a loss on the Gram matrix of the $P$ patch features of an image. With $X_S$ and $X_G$ the $P \times d$ matrices of $\ell_2$-normalized patch features of the student and of a Gram teacher, an early checkpoint of the EMA teacher,

$$\mathcal{L}_{\mathrm{Gram}} = \big\lVert X_S X_S^\top - X_G X_G^\top \big\rVert_F^2 .$$

The loss constrains only the similarities between patches, so the features themselves can keep changing. It starts after 1M iterations, and the Gram teacher is refreshed every 10k iterations; feeding the Gram teacher images at twice the resolution and downsampling its feature maps added about 2 mIoU on ADE20K (§4).

Post-training. A high-resolution phase with global crops of 512 or 768 pixels and Gram anchoring; distillation of the 7B model into ViT-S, S+, B, L and H+ (840M) and ConvNeXt Tiny to Large, several students trained in parallel sharing one teacher; and a text encoder aligned with the frozen ViT-L (§5).

Satellite model. A separate ViT-7B trained on SAT-493M, 493 million $512 \times 512$ crops of Maxar RGB imagery at 0.6 m resolution, and distilled into a ViT-L (§8).

The DINOv3 report is an arXiv technical report; as of October 2026 it has a single version (v1, 13 August 2025).

Other Models Named DINO

DINO (Zhang et al., 2023) is a supervised end-to-end detector in the DETR family that adds contrastive denoising training, mixed query selection and a “look forward twice” box update; Grounding DINO extends that detector with text prompts for open-set detection. Neither involves self-distillation, a momentum teacher or the models on this page. DINOv3’s own detection experiments use Plain-DETR, not the DINO detector, on top of the frozen backbone.

Implementations

All three generations have official PyTorch code from Meta, with weights loadable through torch.hub. As of October 2026:

• DINO: facebookresearch/dino, released under the Apache License 2.0 according to its README, with the ViT-S, ViT-B and ResNet-50 checkpoints. GitHub marks the repository as archived; its last commit dates from April 2023.
• DINOv2: facebookresearch/dinov2. Its README states that DINOv2 code and model weights are released under the Apache License 2.0, as does the TMLR paper (§5). Some later additions to the same repository are licensed differently: according to the README, the XRay-DINO weights and the Cell-DINO weights are under the FAIR Noncommercial Research License and the Cell-DINO code under a CC BY-NC licence. The repository also provides the register variants, linear classification, segmentation and depth heads, and dino.txt text alignment.
• DINOv3: facebookresearch/dinov3. Code and weights are released under the DINOv3 License, a custom Meta licence (last updated 19 August 2025). It grants a royalty-free licence to use, modify and redistribute, with conditions: redistribution under the same agreement, acknowledgement in publications, compliance with trade controls, and a prohibition on uses such as military applications or weapons development. Weights are obtained by submitting a request form, after which download links are sent by email; the README lists the web (LVD-1689M) and satellite (SAT-493M) backbones, ConvNeXt variants, task adapters and the text-aligned model. The README also reports support in the Hugging Face Transformers library from version 4.56.0 and in timm from version 1.0.20.

The repository licences are summarized here, not interpreted; read the licence files for the terms that apply to a given use.