Computer Vision Foundation Models

A computer vision foundation model is a large network pretrained once on broad visual data, such as web images, image–text pairs or masks, and then adapted to many downstream tasks: by prompting it, by training a small head on its frozen features, or by fine-tuning it. Other names in use are vision foundation model, large pretrained model and general-purpose model. The term was introduced in 2021 by a Stanford report that covered language, vision and other modalities; in vision it is applied to image–text encoders such as CLIP, self-supervised encoders of the DINO family such as DINOv2 and DINOv3, the promptable segmenter SAM, open-set detectors, vision–language models and large image generators. This overview covers what the term means and what is disputed about it, the main families, how such models are adapted, evaluated and scaled, and their limits and risks; each family is summarized briefly and linked to the article that explains it.

Definition

Bommasani et al. (2021, §1) of Stanford’s Center for Research on Foundation Models define a foundation model as “any model that is trained on broad data”, generally with self-supervision at scale, that can be adapted, for example by fine-tuning, to a wide range of downstream tasks. The report, posted to arXiv on 16 August 2021 and revised to its third version in July 2022, named BERT, GPT-3 and CLIP as current examples, and its abstract adds DALL-E.

Three points of the definition matter for vision:

• It is functional, not architectural. Nothing in it names a network type, a parameter count or a training objective. A model qualifies by how it is trained (broad data, at scale) and used (adapted to many tasks).
• The model is incomplete by design. The authors chose the word “foundation” because such a model is itself unfinished and serves as the common basis from which task-specific models are built by adaptation (§1.1.1).
• What is new is scale, not technique. The report states that foundation models are based on deep neural networks and self-supervised learning, both decades old, and that transfer learning makes them possible while scale makes them powerful (§1, §1.1).

What is disputed about the term

The name was a deliberate choice among alternatives, and the report records the reasoning (§1.1.1). The authors judged that pretrained model and self-supervised model capture the technical dimension but not the shift in practice they wanted to name, language model too narrow, general-purpose model and multi-purpose model silent about the need for adaptation, and task-agnostic model silent about the consequences for downstream applications. They also intended “foundation” to connote that stability, safety and security matter, and wrote that they could not yet say whether the foundation these models provide is trustworthy.

Vision authors adopted the term with reservations. The Segment Anything paper describes pretrained models as having been adapted to downstream tasks since the early days of machine learning and says they have recently been “(re-)branded” as foundation models; it says SAM fits the definition but notes that a segmentation model covers only a fraction of computer vision, and that, unlike the report’s emphasis on self-supervision, most of SAM’s abilities come from supervised training on masks produced by its data engine (Kirillov et al., 2023, arXiv §8). In our reading, two questions remain open in practice: how broad the data and the set of downstream tasks must be before a pretrained model counts, and whether self-supervision is part of the definition or only its most common route. Papers apply the label to self-supervised encoders (the DINOv3 report presents DINOv3 as a vision foundation model), to promptable models trained mainly with supervision (the SAM 2 paper describes SAM 2 as one for promptable segmentation in images and video) and to generative models, so the term describes a role more than a method.

Intuition

Through the 2010s the standard recipe in vision was to pretrain a network on ImageNet classification and fine-tune it on each new task, the transfer learning paradigm that the report’s vision section describes as the idea that remains at the core of foundation models (Bommasani et al., 2021, §2.2). The report locates the limit of that recipe in its labels: reliance on expensive, carefully collected annotations limits the robustness, generalization and applicability of supervised models.

Foundation models remove most of the labelling bottleneck and then scale. They learn from signals that come with the data: the alt-text next to a web image, another crop of the same photograph, the masked patches of an image, or masks produced by a model-assisted data engine. With labels no longer the bottleneck, data, model size and compute can grow by orders of magnitude, and the resulting representation is general enough that many tasks need only a thin adaptation on top.

A useful picture, in our framing, is a network trained once at high cost and then reused cheaply many times. One stored set of weights becomes the starting point for classification, detection, segmentation, depth estimation, retrieval and captioning, each reached by a different and much cheaper adaptation. The report summarizes the consequences in two words, emergence and homogenization (§1.1). Emergence means that a system’s behaviour is induced by training rather than explicitly built, so capabilities appear without being designed and are hard to anticipate; the report’s example is in-context learning in GPT-3. Homogenization means that many applications are built from the same few models: improvements spread to every adapted model, and so do defects.

Formal Definition

The report’s definition is verbal. The following notation is ours and makes the two stages explicit.

Pretraining. Parameters $\theta$ are fitted once on a broad dataset $\mathcal{D}_{\text{pre}}$ with an objective that refers to no downstream task:

$$\theta^\star = \arg\min_\theta \; \mathbb{E}_{x \sim \mathcal{D}_{\text{pre}}} \, \mathcal{L}_{\text{pre}}(x; \theta),$$

where $x$ may be an image, an image–text pair or an image with masks, and $\mathcal{L}_{\text{pre}}$ is, for example, a contrastive image–text loss, a self-distillation loss, a masked-reconstruction loss, a mask loss or a denoising loss.

Adaptation. For each downstream task $k$, an adaptation procedure $A$ turns $\theta^\star$, a task dataset $S_k$ (possibly empty) and a prompt $\pi_k$ (possibly empty) into a task model:

$$f_k = A\big(\theta^\star,\, S_k,\, \pi_k\big).$$

The adaptation methods differ in what they change. Zero-shot prompting uses $S_k = \emptyset$ and changes nothing in $\theta^\star$; the task is specified by $\pi_k$, such as class names embedded by a text encoder or a click on an object. A linear probe trains only a new head $W$ on $\phi_{\theta^\star}(x)$, the frozen features. Fine-tuning moves $\theta$ itself, starting from $\theta^\star$. Parameter-efficient methods freeze $\theta^\star$ and train a small added set $\Delta$ with $|\Delta| \ll |\theta|$.

On this reading, a model is a foundation model when one $\theta^\star$ serves many tasks $k$, with adaptation much cheaper than pretraining. The definition has no threshold for “many” or “much cheaper”, which is one source of the disagreement over which models qualify.

Properties

Scale of data. The pretraining sets of the models discussed here range from about 142 million curated images (DINOv2’s LVD-142M) through 400 million image–text pairs (CLIP) and over 1.8 billion noisy alt-text pairs (ALIGN) to 1.7 billion curated images selected from about 17 billion (DINOv3), with SAM’s SA-1B holding 1.1 billion masks on 11 million images. Zhai et al. (2022) trained ViTs on the proprietary JFT-3B, nearly three billion images labelled semi-automatically with about 30,000 classes.

Scale of compute. Training runs are measured in thousands to hundreds of thousands of GPU-hours; the section on limits below gives the figures the authors report.

General-purpose objectives. The pretraining loss does not name the downstream tasks. When it does use annotations, as SAM’s mask loss does, the task is chosen to be general (any object, any prompt), not a fixed label set.

Cheap adaptation. The models are released, or meant to be used, as frozen backbones, with prompts, probes or small adapters on top; DINOv2’s authors state the goal as features that work across image distributions and tasks without finetuning (Oquab et al., 2024, abstract).

Composition. Foundation models are often used as components of larger systems. The SAM authors cite CLIP’s use inside DALL·E as an example and designed SAM so that other systems could drive it with prompts (Kirillov et al., 2023, arXiv §8).

Families

Contrastive image–text encoders

CLIP (Radford et al., 2021) trains an image encoder and a text encoder so that matching image–caption pairs from 400 million web pairs land close together in a shared embedding space; class names written as text then become a zero-shot classifier. ALIGN (Jia et al., 2021), from Google, posted to arXiv about two weeks earlier and published at the same ICML, used a similar dual-encoder contrastive recipe on over one billion (1.8B) image–alt-text pairs that had only simple frequency-based filtering, and concluded that the scale of the corpus can make up for its noise. SigLIP (Zhai et al., 2023) replaced the softmax over all pairs in a batch with an independent sigmoid loss per image–text pair, which removes the need for a global normalization over the batch; the authors found that the benefit of larger batches diminished quickly and that a batch of 32k was sufficient. Image–text pretraining also comes in captioning and fusion forms, described under vision–language pretraining. By the SigLIP authors’ account, contrastive image–text pretraining was becoming the usual way to obtain a generic vision backbone, replacing pretraining on large labelled multi-class datasets (Zhai et al., 2023, §1).

Self-supervised image encoders

Self-supervised encoders learn from images alone (self-supervised learning surveys the methods). The masked autoencoder (He et al., 2022) hides most of an image’s patches and reconstructs them; its features need fine-tuning more than probing. The DINO family trains a student network to match an exponential-moving-average teacher on different crops of the same image. DINOv2 (Oquab et al., 2024) combined that objective with masked-patch prediction, curated a 142-million-image dataset by retrieval from web images, and trained a 1.1-billion-parameter ViT-g, distilled into smaller models whose frozen features, its authors report, surpass OpenCLIP’s on most of their image- and pixel-level benchmarks. DINOv3 (Siméoni et al., 2025) grew the teacher to a 6.7-billion-parameter ViT-7B and added Gram anchoring to keep dense patch features from degrading during long training. Its authors argue that self-supervision, needing no metadata, can use virtually unlimited raw images (§1), and the DINOv3 paper cites histopathology, biology, medical imaging, remote sensing and astronomy among the domains in which DINOv2 had been applied. A self-supervised encoder has no text interface; DINOv3 adds one after training by aligning a text encoder to the frozen image encoder (§5.3).

Promptable segmentation

SAM (Kirillov et al., 2023) defines a task rather than a label set: given an image and a prompt (a point, a box, a rough mask), return a valid mask, even when the prompt is ambiguous. A large ViT encodes the image once and a light decoder answers each prompt in real time. Because masks are not abundant on the web, the authors built a data engine in which the model helped annotators and then annotated alone, producing SA-1B, 1.1 billion masks on 11 million licensed images. SAM 2 (Ravi et al., 2025) extends the task to video with a streaming memory of past frames, trained with its SA-V dataset of 35.5 million masks across 50.9 thousand videos, among other data; its authors report that on images it is more accurate than SAM and six times faster. The segmentation family is the clearest case of a foundation model trained mainly with supervision, at the scale a data engine makes possible.

Open-set detection and grounding

Detection data are expensive to label, so open-set detectors borrow vocabulary from image–text pretraining. OWL-ViT (Minderer et al., 2022) starts from a contrastively pretrained ViT and text encoder, attaches a box head and a class-embedding head to every output token, replaces fixed class weights with text embeddings of class names, and fine-tunes on detection data; its authors found that more image-level pretraining and larger models gave consistent gains in detection. Grounding DINO (Liu et al., 2024) fuses a BERT text encoder with a DETR-family detector at several stages, so that a prompt listing categories or a referring expression returns boxes. Its “DINO” is the supervised detector of that name, not the self-supervised DINO above. Text-prompted detectors are often chained with promptable segmenters, boxes from one serving as prompts to the other, as in Grounded SAM; visual grounding covers the language side of this problem.

Vision–language models

A vision–language model connects a pretrained vision encoder to a pretrained large language model through a connector, so that captioning, visual question answering and document reading become instructions to one model. LLaVA (Liu et al., 2023) is a simple form: a frozen CLIP ViT-L/14, a projection from each patch feature to a language-model token, and a decoder-only language model fine-tuned on instruction data generated with a text-only GPT-4. In our reading, VLMs are foundation models built from foundation models: the image encoder is usually itself a contrastive or self-supervised encoder of the kinds above, an instance of what the report calls homogenization.

Generative models

The report counts generative models as foundation models; its abstract lists DALL-E among its examples. In vision the most reused design is the latent diffusion model (Rombach et al., 2022), which runs diffusion in the latent space of a pretrained autoencoder and uses cross-attention layers to condition on text or boxes; it underlies Stable Diffusion. Such a model is adapted not only for text-to-image generation and image editing but also for perception: Marigold (Ke et al., 2024) fine-tunes Stable Diffusion into an affine-invariant monocular depth estimator, using only synthetic training data and a couple of days on a single GPU, on the premise that the generator’s prior about the visual world helps depth estimation generalize. Video generation extends the same designs to time.

Adaptation

The report frames the choice of adaptation method by three factors: the compute budget, especially storage and memory; how much task-specific data is available; and how much access the user has to the model’s gradients (Bommasani et al., 2021, §4.3.1). In vision, the main options run from changing nothing to changing everything.

Zero-shot prompting. The task is specified at inference time: class names or descriptions for CLIP, a click or box for SAM, a category list or referring expression for Grounding DINO, an instruction for a VLM. No labelled data are needed, but the result depends on the prompt; CLIP’s own experiments showed that the label set offered to the model changes its errors (see the section on limits).

Linear probing. A linear classifier is trained on frozen features. It is cheap, needs one stored backbone for all tasks and, by design, cannot change the representation, which is why it doubles as an evaluation protocol (representation learning explains it).

Fine-tuning. All or some weights move from $\theta^\star$. It is usually the most accurate in distribution, but Kumar et al. (2022) found that with good pretrained features and a large distribution shift, fine-tuning can be worse than a linear probe out of distribution, and Wortsman et al. (2022) observed that fine-tuning zero-shot models such as CLIP or ALIGN improves target accuracy but often reduces robustness to distribution shifts. Their remedies, probing before fine-tuning (LP-FT) and ensembling the weights of the zero-shot and fine-tuned models (WiSE-FT), are explained in fine-tuning, which also covers layer freezing and layer-wise learning rates.

Parameter-efficient fine-tuning. The backbone is frozen and a small set of new parameters is trained per task, so one copy of a large model serves many tasks. Adapters add a few trainable modules per task to a network whose original parameters stay fixed (Houlsby et al., 2019); LoRA injects trainable rank-decomposition matrices into each layer of a frozen Transformer, with no added inference latency (Hu et al., 2022); and visual prompt tuning trains a small number of parameters in the input space of a frozen vision Transformer, under 1% of the model’s parameters (Jia et al., 2022). The report notes that such lightweight methods sometimes match full fine-tuning while updating about a thousand times fewer parameters, and that it was uncertain how they scale with model size (§4.3.1).

Task heads on frozen features. For dense tasks, a decoder is trained on the frozen backbone’s patch features. DINOv3’s authors evaluate detection and segmentation this way, training only the detection or segmentation components on top of the frozen backbone.

Composition. A model’s outputs become another model’s inputs: boxes from an open-set detector prompt a promptable segmenter, an image encoder feeds a language model, a pretrained generator becomes a depth estimator. In our reading, much of the practical value of foundation models in vision comes from such pipelines, which need no gradient access at all.

As a rule of thumb that the sources support only in part: with few labels and a model that already covers the domain, prompting or probing is the cheap first try; with more labels or a large domain gap, fine-tuning or parameter-efficient fine-tuning; when robustness to shift matters, methods that stay close to the pretrained weights.

Evaluation Across Tasks

A task-specific model is evaluated on its task. A foundation model is an intermediate product, so its quality depends on what it will be adapted to and how. The report separates intrinsic evaluation of the foundation model from extrinsic evaluation of the task models adapted from it, and notes that comparing foundation models is complicated because each must be adapted, possibly in different ways, before it performs any task (Bommasani et al., 2021, §4.4.1).

Protocols. Four protocols are common: zero-shot accuracy with text prompts, linear probing, k-nearest-neighbour classification on frozen features, and fine-tuning. They can rank models differently; representation learning gives examples, including the VTAB finding that linear and fine-tuned results correlated poorly on many specialized and structured tasks. For dense features, DINOv2 and DINOv3 report linear segmentation and depth probes on frozen patch features.

Benchmark suites. The Visual Task Adaptation Benchmark (VTAB; Zhai et al., 2019) defines a good representation as one that adapts to diverse, unseen tasks with few examples: 19 tasks in natural, specialized and structured groups, with 1,000 labelled examples per task in its main setting. Later papers report averages over larger suites, such as the 35-dataset “VTAB+” collection that Cherti et al. (2023) used for zero-shot evaluation. A suite average hides the per-task spread that VTAB was designed to expose.

Robustness to distribution shift. Image classifiers are often tested on datasets built to share ImageNet’s classes but not its distribution. The CLIP paper used the seven natural distribution shifts studied by Taori et al.: ImageNetV2, ImageNet Sketch, Youtube-BB, ImageNet-Vid, ObjectNet, ImageNet Adversarial (ImageNet-A) and ImageNet Rendition (ImageNet-R). It adopted their distinction between effective robustness, accuracy under shift above what in-distribution accuracy predicts, and relative robustness, any improvement under shift (Radford et al., 2021, arXiv §3.3). Zero-shot CLIP models reduced the gap between ImageNet accuracy and accuracy under shift by up to 75%. The best zero-shot CLIP and a ResNet-101 with the same 76.2% ImageNet accuracy scored 88.9% and 37.7% on ImageNet-R, and 72.3% and 32.6% on ObjectNet (Figure 13). The same paper warns against reading this too simply. Adapting CLIP to ImageNet with a logistic-regression classifier raised ImageNet accuracy by 9.2 points, to 85.4%, while average accuracy under the shifts slightly decreased, and the authors could not say whether the robustness came from zero-shot use, the size and diversity of the pretraining data or natural-language supervision.

Contamination. Web-scale pretraining sets may contain copies of evaluation images. CLIP’s authors ran a duplicate detector on 35 evaluation datasets and found a median overlap of 2.2% and a mean of 3.2%, which rarely shifted accuracy by more than 0.1%; they also noted that they could not check the detector’s recall across 400 million training images (arXiv §5). DINOv2 removed near-duplicates of benchmark test and validation images from its data pipeline. Curating data by retrieval toward benchmark domains, as DINOv2 did, raises a related question: in our reading, part of the transfer then comes from data chosen to resemble the evaluation domains. Benchmark overfitting discusses the general problem.

What the benchmarks leave out. The report adds that standard metrics for generated images, such as the Fréchet Inception Distance, have known flaws, and that human evaluation is costly (§2.2.2).

Scaling

Scaling studies fit how error falls as model size, data and compute grow. Zhai et al. (2022), studying ViTs pretrained on up to three billion labelled images, found that error against compute follows a power law over more than two orders of magnitude but saturates at both ends, and fitted a double-saturating power law

$$E = a\,(C + d)^{-b} + c ,$$

where $C$ is compute, $c$ is a floor that even unlimited capacity does not pass, and $d$ accounts for the non-zero accuracy of trivial models at the low end (§2.2). They also found that representation quality can be bottlenecked by model size, that small models do not benefit from more data, that the largest models still improved when the data grew from one to three billion images, and that bigger models are more sample-efficient. Their two-billion-parameter ViT-G/14 reached 90.45% top-1 on ImageNet after fine-tuning and 84.86% in 10-shot linear evaluation (10 examples per class).

For image–text models, Cherti et al. (2023) trained CLIP models on the public LAION data with the open-source OpenCLIP code, on up to two billion pairs, and found power-law scaling for zero-shot classification, retrieval, linear probing and fine-tuning. They also found that OpenAI’s CLIP and OpenCLIP models scaled differently despite identical architectures and similar recipes, which they attribute to the training distribution, so a scaling law describes a dataset as well as an architecture. OWL-ViT’s detection results kept improving with model size and pretraining duration beyond 20 billion image–text pairs (Minderer et al., 2022, arXiv §1).

Scale is not the only variable: DINOv2’s curated 142 million images beat as many uncurated ones on most benchmarks (Oquab et al., 2024, §6.2), and SigLIP’s batch-size gains saturated (above). The report notes that scale itself rests on hardware improvements, the transformer architecture and the availability of data (Bommasani et al., 2021, §1.1). In our reading, the scaling results describe trends within a training recipe; they do not by themselves say which recipe to choose.

Limits and Risks

Data provenance and licences

Most pretraining data come from the web, and their provenance determines what may be done with the model. Large sets are often private: Zhai et al. (2022) used the proprietary JFT-3B, and Cherti et al. (2023) describe earlier scaling-law work as having mainly used private data and models. The open alternative, LAION, is a list of image URLs and alt-text from Common Crawl; its datasheet states that people in the images were not notified of the collection, LAION’s own position is that the images remain under their owners’ copyright, and the dataset was withdrawn in December 2023, when a Stanford Internet Observatory report identified links to suspected child sexual abuse material, and re-released as Re-LAION-5B in August 2024. The LAION article gives the details and the sources. SAM’s authors describe SA-1B as built from licensed, privacy-respecting images, with faces and licence plates blurred in the released data. DINOv3’s main dataset was selected from about 17 billion Instagram images that had passed the platform’s moderation.

Weight licences vary as well. As of October 2026, the repository READMEs state that the code and weights of SAM, SAM 2 and DINOv2 are released under the Apache License 2.0, that some later additions to the DINOv2 repository use non-commercial licences, and that DINOv3’s code and weights are released under a custom DINOv3 License. Licence terms apply to the weights, not to the data the model was trained on.

Bias

A model trained on web data learns the web’s associations. The CLIP paper probed this directly: when zero-shot CLIP classified 10,000 FairFace images with added labels including “animal”, “gorilla”, “thief” and “criminal”, 4.9% of the images were classified into the non-human classes, about 14% of images in the FairFace category Black against under 8% for every other race category, and 16.5% of men’s images against 9.8% of women’s fell into crime-related classes (Radford et al., 2021, arXiv §7.1). The label set mattered: adding a “child” class cut the share of images of children aged 0 to 2 assigned to crime-related or non-human classes from 30.3% to 2.3% (Table 7). Zero-shot classification thus inherits both the biases of the training data and those of whoever writes the label list. Dataset bias covers how such bias arises and is measured, and the LAION article documents audits of web-scale image–text data.

Compute, energy and carbon

Only some papers report their compute, and the figures below are the authors’ own estimates under their stated assumptions; the DINO reports assume a power usage effectiveness of 1.1, a US-average grid of 0.385 kg CO₂eq per kWh and the GPUs’ thermal design power, and count only the GPUs’ electricity, not manufacturing or disposal (DINOv3 also names cooling among the excluded emissions).

Model	Reported training compute	Energy and emissions as reported	Source
CLIP RN50x64	18 days on 592 V100 GPUs	not reported	Radford et al. (2021), §2.5
CLIP ViT-L/14	12 days on 256 V100 GPUs	not reported	Radford et al. (2021), §2.5
DINOv2 ViT-g	22,016 A100 GPU-hours	9.7 MWh, 3.7 t CO₂eq	Oquab et al. (2024), Table 14
MetaCLIP ViT-G (DINOv3 authors’ estimate)	368,640 A100 GPU-hours	160 MWh, 62 t CO₂eq	Siméoni et al. (2025), Table 20
DINOv3 ViT-7B	61,440 H100 GPU-hours	47 MWh, 18 t CO₂eq	Siméoni et al. (2025), Table 20
SAM 2 (released model)	108 hours on 256 A100 GPUs	12,165 kWh, 3.89 t CO₂eq	Ravi et al. (2025), arXiv v2, model card (Appendix H.1)

A final training run is a small part of a project. DINOv2’s authors estimated their whole project at 0.5k to 1k t CO₂eq, about 200k GPU-days, and DINOv3’s at roughly 2,600 t CO₂eq from about 9 million GPU-hours. Comparing retraining costs under the same assumptions, DINOv2’s authors estimated that OpenCLIP ViT-L and ViT-G would need 22.4 and 118.9 MWh, against 9.7 MWh for DINOv2, and concluded that for visual features alone self-supervised training is preferable in carbon terms, while acknowledging that the comparison is unfair because the CLIP models also train a text encoder (Oquab et al., 2024, §9). The report adds that deployment at scale can consume substantial energy beyond training (Bommasani et al., 2021, §5.3). For a user, the trade-off runs the other way: adapting a released model costs a small fraction of pretraining. DINOv2’s authors put a fine-tuning on ImageNet-1k at about 1k GPU-hours and 0.2 t CO₂eq, against 3.7 t for a ViT-g pretraining.

Homogenization

The report’s central warning is that when many systems are adapted from a few models, any flaws in those models are inherited by all adapted models, giving leverage and single points of failure at once (Bommasani et al., 2021, §1.1). In vision the pattern is visible: VLMs such as LLaVA build on CLIP encoders, Grounding DINO pipelines feed SAM, and DINOv2 and DINOv3 are offered as frozen backbones for medical, satellite and natural images alike. A bias, a blind spot or a licence restriction in the base model then reaches every application built on it.

Opacity and unexpected failures

Because their abilities emerge from training, the report describes foundation models as hard to understand and prone to unexpected failure modes (§1.1). The methods of neural network interpretability apply to them, at greater cost. The report’s vision section also records that current models of the time still struggled to generalize to compositions of simple shapes and colors (§2.2.2). Vision–language models add failures of their own, such as describing objects that are not in the image (multimodal hallucination).

Common Misconceptions

• “A foundation model is a particular architecture.” The definition is about training and use. ViTs dominate in practice, but DINOv3 also distills into ConvNeXt students, and the original latent diffusion model’s denoiser is a U-Net (Rombach et al., 2022).
• “Foundation models are always self-supervised.” The report says “generally”, and SAM’s authors stress that most of their model’s abilities come from supervised training on masks.
• “Zero-shot means the model never saw anything like the test data.” It means no training on the target task’s labelled examples. Web-scale pretraining may contain similar images or even copies of test images, which is why CLIP measured overlap.
• “Bigger is always better.” Scaling studies report saturation, bottlenecks from model size, and laws that change with the training distribution; data curation and batch size have their own limits.
• “Fine-tuning is always the best adaptation.” It is usually best in distribution, but it can lose robustness under shift, and it requires a full copy of the model per task.
• “An open licence makes the model unencumbered.” Weight licences differ from model to model, and the provenance of the training data raises separate questions.

Where It Is Used

The families above reach most of the tasks on this site. Image classification and image–text retrieval use image–text encoders zero-shot or with probes. Object detection, semantic segmentation and instance segmentation build on pretrained backbones, open-set detectors and promptable segmenters. Monocular depth estimation uses DINOv2-initialized encoders and repurposed generators, and 3D reconstruction has large pretrained models of its own. Image captioning and visual question answering run on vision–language models, and image generation and editing build on latent diffusion models. Deep learning places foundation models in the history of the field, and multimodal learning covers models that combine vision with other modalities.