Medical Foundation Models

A medical foundation model is a large network pretrained once on broad medical imaging data, such as millions of unlabelled tissue tiles, retinal photographs, figure–caption pairs from the biomedical literature or image–mask pairs, and then adapted to many clinical imaging tasks by prompting it, by training a small head on its frozen features, or by fine-tuning it. The idea is that of the general computer vision foundation models, applied where labels are scarcer and more expensive than in natural images: unlabelled medical images accumulate in hospital archives, while expert annotation does not scale. Other names in use are pathology foundation model, ophthalmic foundation model, biomedical vision–language model and generalist medical model. This article covers what is pretrained and how, the main cases in pathology, radiology, ophthalmology and segmentation, how such models are evaluated, the contamination and benchmark problems that complicate the comparisons, and what the evidence on clinical use shows so far. It describes methods and evidence for practitioners and researchers; it gives no clinical guidance. The area changes quickly: the facts below are as the cited papers and pages state them, as of November 2026.

Definition

The papers in this area adopt the general definition. RETFound’s authors define a foundation model as a large model trained on a vast quantity of unlabelled data at scale, so that it can be adapted to a wide range of downstream tasks (Zhou et al., 2023), and UNI’s authors present their general-purpose encoder as a foundation model for developing AI models in anatomic pathology (Chen et al., 2024, arXiv version). What makes a model medical is its pretraining data and its intended tasks, not its architecture: the networks are the vision transformers, CNNs and image–text encoders of general vision, often initialized from natural-image weights.

Three points of usage matter in practice:

• The label “foundation” is applied to different kinds of model. A self-supervised tile encoder (UNI), an image–text model with a zero-shot interface (CONCH, BiomedCLIP), a fine-tuned promptable segmenter (MedSAM) and a slide-level encoder (Prov-GigaPath) are all described by their authors as foundation models, as in general vision, where the term describes a role more than a method (foundation models discusses the dispute).
• Breadth is relative. Most medical foundation models cover one modality or one specialty: H&E pathology, colour fundus photographs and OCT, chest radiographs. Image–text models trained on figures from the literature, such as BiomedCLIP, and MedSAM, trained on ten imaging modalities, are broader.
• “Foundation” is a claim about adaptation, not about clinical readiness. None of the model papers below presents its foundation model as a clinical device; clinical use requires a task model adapted from it, validated for that use (clinical validation).

Intuition

The task-specific recipe that dominated medical imaging through the 2010s starts each project from labelled data: collect images, have experts annotate them, and train a network initialized from ImageNet weights by transfer learning. Each new task repeats the annotation, and the networks it produces transfer poorly beyond the hospitals and scanners they were trained on (Medical Computer Vision gives examples of that domain shift).

A foundation model moves most of the learning to a stage that needs no task labels. RETFound’s authors put the motivation in terms of annotation: expert labels are scarce, while large volumes of retinal images accumulate unlabelled in routine care (Zhou et al., 2023). Pretraining on those images, with self-supervised objectives or with text that comes with them, produces features that already separate tissue types, lesions and anatomical structures, and a task then needs only a thin layer of supervision on top. The hope, which the evaluations below test, is twofold: fewer labels for the same accuracy (label efficiency), and better transfer to new sites (generalization).

A useful picture, in our framing, is a pathology department with a million archived slides and labels for a few hundred of them for any one question. Training a network from the few hundred labelled slides uses a tiny fraction of the available images; pretraining on the archive first lets the labelled slides refine a representation instead of building one.

Formal Definition

The notation in this section is ours; the objectives are those of the cited papers.

Pretraining. An encoder $\phi_\theta$ is fitted once on a pretraining set $\mathcal{D}_{\text{pre}}$ that carries no labels for the downstream tasks:

$$\theta^\star = \arg\min_\theta \; \mathbb{E}_{x \sim \mathcal{D}_{\text{pre}}}\, \mathcal{L}_{\text{pre}}(x;\theta).$$

Three objectives cover the cases in this article.

Masked image modelling hides a large fraction of an image’s patches and reconstructs them. With a binary mask $m_p = 1$ for visible and $0$ for hidden pixels $p$, and a decoder producing $\hat{x}$ from the visible part, the loss is the reconstruction error over hidden pixels,

$$\mathcal{L}_{\text{MIM}} = \frac{\sum_p (1-m_p)\,(\hat{x}_p - x_p)^2}{\sum_p (1-m_p)} .$$

RETFound uses the masked autoencoder with mask ratios of 0.75 for fundus photographs and 0.85 for OCT (Zhou et al., 2023, Methods). UNI uses DINOv2, which combines a masked-image-modelling objective with self-distillation between a student and a teacher network (DINO explains the method; Chen et al., 2024, arXiv version).

Image–text contrastive learning trains an image encoder $f$ and a text encoder $g$ so that, in a batch of $N$ pairs $(x_i, t_i)$, matching pairs have high cosine similarity $s_{ij} = \langle f(x_i), g(t_j) \rangle / (\lVert f(x_i)\rVert \lVert g(t_j)\rVert)$ and non-matching pairs low similarity. In the symmetric form used by CLIP, with a temperature $\tau$,

$$\mathcal{L}_{\text{ITC}} = -\frac{1}{2N}\sum_{i=1}^{N}\left[\log \frac{e^{s_{ii}/\tau}}{\sum_{j} e^{s_{ij}/\tau}} + \log \frac{e^{s_{ii}/\tau}}{\sum_{j} e^{s_{ji}/\tau}}\right].$$

CheXzero follows CLIP’s training procedure on chest radiographs and reports (Tiu et al., 2022); CONCH adds a captioning objective in the CoCa design (Lu et al., 2024, arXiv version).

Promptable segmentation trains an image encoder $f$, a prompt encoder $e$ and a mask decoder $h$ so that $\hat{M} = h(f(x), e(b))$ matches the mask for the object indicated by a prompt $b$. MedSAM uses a bounding box as the prompt and the sum of binary cross-entropy and Dice loss (Ma et al., 2024, Methods, equations 1–3).

Adaptation. For a task $k$ with a labelled set $S_k = \{(x_i, y_i)\}$, the main options differ in what they train:

• A linear probe trains only a classifier on frozen features. UNI’s authors fit $\ell_2$-regularized logistic regression with the coefficient fixed at $\lambda = 100/(MC)$, where $M$ is the embedding dimension and $C$ the number of classes, solved with L-BFGS (Chen et al., 2024, arXiv version, Methods).
• Nearest-neighbour and prototype probes train nothing. In the SimpleShot form UNI’s authors use for few-shot evaluation, each class $c$ gets a prototype, the mean of its $K$ support features, $\mu_c = \frac{1}{K}\sum_{i: y_i = c} \phi_{\theta^\star}(x_i)$, and a test image goes to the class with the nearest prototype in Euclidean distance.
• Zero-shot prompting uses no labelled examples. CheXzero scores each finding by comparing the image with a positive and a negative prompt, such as “consolidation” and “no consolidation”, through a softmax over the two logits, $p = e^{s^+/\tau}/(e^{s^+/\tau} + e^{s^-/\tau})$, rather than normalizing across findings as CLIP does across classes (Tiu et al., 2022, Methods).
• Fine-tuning updates $\theta$ from $\theta^\star$ with a task head, as RETFound does for every downstream task.

Properties

What is pretrained and on what. The pretraining sets are large by medical standards and small by web standards. As the authors report them: UNI, over 100 million tissue images from 100,426 H&E slides; RETFound, 904,170 colour fundus photographs and 736,442 OCT scans; CONCH, 1.17 million histopathology image–caption pairs; BiomedCLIP, 15 million figure–caption pairs; CheXzero, 377,110 chest radiograph–report pairs; MedSAM, 1,570,263 image–mask pairs. Most come from one or a few institutions; RETFound’s authors, for example, list the predominance of UK cohorts in their data as a limitation (Zhou et al., 2023, Discussion).

Initialization from natural images. Most medical foundation models do not start from random weights. RETFound continues masked-autoencoder pretraining from weights already pretrained on ImageNet-1k, and its authors report that this sequence beat self-supervised pretraining on retinal images alone (Zhou et al., 2023); CheXzero starts from OpenAI’s CLIP weights (Tiu et al., 2022); MedSAM starts from SAM’s ViT-Base (Ma et al., 2024).

Cost is front-loaded. Pretraining is the expensive step and adaptation the cheap one. RETFound’s pretraining took two weeks on eight NVIDIA A100 GPUs, and adapting it to a task with 1,000 images took about 1.2 hours on a single NVIDIA T4 (Zhou et al., 2023, Discussion). The asymmetry is the practical argument for releasing pretrained weights: a group without a large archive can adapt a model it could not have trained.

Interfaces differ by family. A self-supervised encoder offers only features, so every task needs labelled data, if few. An image–text model also offers a text interface, so class names become a zero-shot classifier and text queries retrieve images. A promptable segmenter offers an interactive interface, a box or a click per object.

Pretraining Families

Self-supervised image encoders

Self-supervised encoders learn from images alone and are the most common kind. In pathology, UNI is a ViT-Large trained with DINOv2 on Mass-100K, a set built from in-house slides of two Boston hospitals and from the GTEx consortium’s autopsy slides. Its authors deliberately left public collections such as TCGA, CPTAC and PAIP out of pretraining so that those could serve for external evaluation (Chen et al., 2024, arXiv version). Digital Pathology compares UNI with Virchow and Prov-GigaPath, and Whole-Slide Image Classification shows how such tile features feed slide-level models.

In ophthalmology, RETFound is two ViT-Large encoders, one for colour fundus photographs and one for OCT, pretrained as masked autoencoders on images from MEH-MIDAS, the imaging records of 37,401 patients with diabetes seen at Moorfields Eye Hospital, London, between 2000 and 2022, together with public data; 9.8% of the fundus photographs came from Kaggle EyePACS (Zhou et al., 2023). The authors replaced the masked autoencoder with four contrastive methods (SimCLR, SwAV, DINO and MoCo-v3) inside the same framework and found the masked autoencoder better on most tasks, while cautioning that architectures and hyperparameters also differed between these variants (Zhou et al., 2023, Discussion). Ophthalmic Image Analysis places RETFound among retinal methods.

In radiology, RAD-DINO’s authors argue against relying on text: radiology reports describe specific findings, so features learned from them are limited by what the reports mention. According to their abstract, their encoder, pretrained only on imaging data, matched or exceeded biomedical image–text models on classification, segmentation and report generation, and its features correlated better than text-supervised ones with attributes that reports rarely mention, such as sex or age (Pérez-García et al., 2025, abstract).

Image–text models

Image–text models learn from images paired with captions or reports. The pairing gives them the zero-shot interface of CLIP, at the cost of learning only what the text describes.

CheXzero (Tiu et al., 2022) trained on 377,110 chest radiograph–report pairs from MIMIC-CXR, initialized from CLIP’s ViT-B/32 and text Transformer. On the CheXpert test set of 500 radiographs, its mean AUC over five findings was 0.889 with no labelled training examples; against three board-certified radiologists, the difference in Matthews correlation coefficient averaged over the five findings was not statistically significant ($-0.005$). On PadChest, collected at a different hospital from the training data, it outperformed a fully supervised CheXNet model on three of eight findings. Chest X-Ray Analysis describes these datasets.

CONCH (Lu et al., 2024, arXiv version) trained on 1.17 million histopathology image–caption pairs, cleaned from 1.79 million pairs collected from educational sources and the PubMed Central Open Access dataset and filtered to human tissue. Classes are scored by text prompts; the authors ensemble several prompts per class, which they found to raise performance in most tasks compared with single prompts, except where a model’s median performance was already near chance. In their few-shot comparisons, CONCH used zero-shot outperformed supervised models trained on PLIP or BiomedCLIP features with up to 64 labels per class for invasive breast carcinoma subtyping and up to 128 for non-small-cell lung cancer subtyping.

BiomedCLIP is broader. Its model card describes it as pretrained by contrastive learning on PMC-15M, 15 million figure–caption pairs from biomedical research articles in PubMed Central covering microscopy, radiography, histology and other image types, with PubMedBERT as the text encoder and a vision transformer as the image encoder; as of November 2026 the card lists an MIT licence, notes that the model is English-only, and places any deployed use, commercial or otherwise, out of scope (Microsoft, model card). The paper itself (Zhang et al., 2025) could not be read for this article.

Promptable segmentation models

MedSAM adapts the Segment Anything Model to medical images. Its authors initialized it from SAM’s ViT-Base, kept SAM’s prompt encoder fixed, and trained on 1,570,263 image–mask pairs from 10 imaging modalities and over 30 cancer types; 3D CT and MR volumes are processed as series of 2D slices (Ma et al., 2024). They compared it with SAM and with specialist U-Net and DeepLabV3+ models trained for each modality, which were given the same box as an extra input channel, on 86 internal and 60 external validation tasks, and report better accuracy and robustness than the specialists in the abstract; Medical Image Segmentation gives the comparison and its wording in the results section. The authors list two limitations: CT, MRI and endoscopy dominate the training set, and box prompts are ambiguous for vessel-like branching structures.

Slide-level and fine-tuned models

Pathology adds a level above tiles. Prov-GigaPath pretrains a slide encoder over the sequence of tile embeddings (Xu et al., 2024), described in Whole-Slide Image Classification. Foundation models are also starting points for clinical task models: EAGLE, a predictor of EGFR mutations in lung adenocarcinoma from H&E slides, fine-tunes Prov-GigaPath’s tile encoder on 5,174 slides from Memorial Sloan Kettering Cancer Center (Campanella et al., 2025); its evaluation is described below under clinical use.

Evaluation

A foundation model is an intermediate product, so it is evaluated through the task models adapted from it, and the choice of adaptation is part of the result (foundation models separates intrinsic and extrinsic evaluation).

Protocols

The papers use five protocols, often several in one paper:

Protocol	What is trained	Example in the papers
Linear probe	A linear classifier on frozen features	UNI, CONCH (tile-level tasks)
$k$-NN and prototype probes	Nothing; labelled features are compared	UNI (20-NN, SimpleShot)
Zero-shot prompting	Nothing; class prompts are written	CONCH, CheXzero
Weakly supervised slide model	An aggregator on frozen tile features	UNI, CONCH, Neidlinger et al. (2025)
Fine-tuning	The whole network	RETFound, EAGLE

UNI’s authors describe linear probing as measuring discriminative transfer and $k$-nearest-neighbour probing as measuring representation quality, the latter being non-parametric apart from the choice of $k$ (Chen et al., 2024, arXiv version). Representation Learning explains why the protocols can rank encoders differently.

Few labels

Label efficiency is measured by training with a varying number of labels. UNI’s authors sample $K$ examples per class, for $K$ from 1 to 256 at tile level and from 1 to 32 at slide level, repeat each setting over 1,000 random draws (“episodes”) at tile level and 5 at slide level, and report the distribution as box plots; on their slide-level tasks other than coarse-grained breast subtyping (BRACS), the next best encoder needed up to eight times as many labelled slides per class to match UNI’s median 4-shot performance (Chen et al., 2024, arXiv version). RETFound’s authors trained with fractions of the training set and report that RETFound outperformed the comparison models on heart-failure prediction with 10% of the labelled data (Zhou et al., 2023). With few labels, results vary greatly with the particular examples drawn, which is why repeated draws are needed; the example below shows the size of that spread on synthetic images.

Many tasks

A general-purpose claim needs many tasks. UNI’s evaluation covers 33 tasks in the arXiv version (34 in the published abstract), including tissue classification, retrieval, segmentation and slide classification of up to 108 cancer types; CONCH’s suite counts 13 benchmarks in the arXiv version and 14 in the published abstract; RETFound’s covers ocular disease diagnosis on eight public datasets, prognosis of conversion to neovascular age-related macular degeneration and three-year incidence of four systemic diseases. An average over tasks hides the per-task spread, and each group chooses its own tasks (Digital Pathology notes that their numbers are not directly comparable).

Comparisons and external evaluation

What a model is compared against shapes what a result shows. UNI’s comparisons are with an ImageNet ResNet-50 and earlier pathology encoders (CTransPath, REMEDIS); RETFound’s with the same architecture pretrained by supervised learning on ImageNet-21k, by self-supervised learning on ImageNet-1k, or by self-supervised learning on retinal images alone; MedSAM’s with SAM and with specialist models trained per modality. Only the last compares against specialist models trained from scratch on the target data, per modality and, for four tasks, per task.

External evaluation tests transfer. RETFound’s authors cross-evaluated diabetic retinopathy models between APTOS-2019, IDRiD and MESSIDOR-2, and evaluated systemic disease prediction on UK Biobank after training on Moorfields data; performance fell markedly, for instance by 0.16 AUROC with fundus photographs and 0.19 with OCT for ischaemic stroke, which they attribute to differences in age, ethnicity and imaging devices, while RETFound stayed ahead of the comparison models in most external tasks (Zhou et al., 2023, Discussion).

Contamination and Benchmark Problems

Several problems make reported numbers harder to interpret than in task-specific work.

Overlap between pretraining and test data. An encoder pretrained on the slides or images later used for testing has seen the test distribution, and possibly the test images, without labels. UNI’s authors call data contamination a growing concern for foundation models trained on large public collections, noting that evaluation in which the model has seen test data during training may be optimistically biased; because CTransPath and REMEDIS were pretrained on almost all TCGA slides, they treat those models’ TCGA results as possibly inflated (Chen et al., 2024, arXiv version). Whole-Slide Image Classification gives the slide-level comparison they made with this in mind.

Literature-derived pretraining data. CONCH and BiomedCLIP learn from figures published in the biomedical literature. In our reading, this raises a question their evaluations have to answer: papers describing public benchmark datasets often reproduce images from those datasets, so figure–caption corpora may contain benchmark images. We did not find this examined in the sources read for this article; it is a question to ask of any image–text model built from published figures.

Self-selected benchmarks. Each group compares its model with earlier ones on tasks of its own choosing, and in each of the papers above the authors’ own model comes out ahead overall. Independent benchmarks change the picture. Neidlinger et al. (2025) evaluated 19 pathology foundation models on 31 weakly supervised tasks (morphology, biomarkers and prognosis) in 13 cohorts with 6,818 patients and 9,528 slides, training on TCGA features and testing on external cohorts, including proprietary cohorts from several countries that, they state, were never part of any foundation model’s training. They describe the narrow benchmarks of earlier evaluations as risking data leakage and selective reporting. CONCH and Virchow2 reached the highest mean AUROC across all tasks, 0.71, followed by Prov-GigaPath and DinoSSLPath at 0.69. CONCH’s advantage was less pronounced with little training data and low-prevalence tasks, an ensemble of CONCH and Virchow2 outperformed the single models in 55% of tasks, and correlations of downstream performance with pretraining-set size and diversity were positive but mostly not statistically significant. A mean AUROC of 0.71 also shows that many of these tasks remain hard for every encoder.

Labels used by “label-free” methods. Zero-shot results can still depend on labelled data. CheXzero’s authors note that their method needed a labelled validation set to select hyperparameters and to set the per-finding thresholds behind their MCC and F1 comparisons with radiologists (Tiu et al., 2022, Discussion). The AUC needs no threshold; the comparisons with readers do.

Prompts derived from the reference. MedSAM’s box prompts were simulated from the expert masks, with a random perturbation of 0 to 20 pixels (Ma et al., 2024, Methods; the statement is in the training protocol, and no separate procedure for validation boxes is described). In our reading, such boxes are tighter and more consistent than those a user would draw, so the reported Dice reflects an idealized prompt; the user study described below measures a different quantity.

Confounders. Retinal images carry age and other signals that correlate with disease. RETFound’s authors tested this for myocardial infarction by varying the age difference between cases and controls: at the largest difference, 5.3 years, logistic regression on age alone reached an AUROC of 0.63, exceeding the ImageNet-pretrained comparison models, while RETFound kept stable performance as the age difference shrank (Zhou et al., 2023). A model can score well by learning a confounder rather than disease; Dataset Bias and Benchmark Overfitting cover the general problems.

Splits. The usual medical leakage rules apply: MedSAM’s authors split CT and MRI by scan and video data by video so that slices or frames of one case never fall on both sides (Ma et al., 2024), and pathology papers split by patient or case (Training, Validation, and Test Sets).

Evidence on Clinical Use

The designs described in Clinical Validation and Regulation of Medical Imaging AI apply unchanged to task models built from foundation models. As of November 2026, the studies read for this article sit at the following levels.

Retrospective evaluation on held-out and external data. This is the level of the model papers themselves: UNI, CONCH, RETFound, CheXzero and MedSAM report retrospective results on held-out splits and external datasets, and Neidlinger et al. (2025) on external cohorts. CheXzero adds a comparison with three radiologists on a fixed 500-image test set; that is a retrospective standalone comparison, not a reader study of radiologists using the model.

Retrospective studies in a target setting. Zhang et al. (2024) fine-tuned RETFound on 6,599 fundus photographs from a community screening programme in Shanghai, where images are taken without pupil dilation and are often of low quality, and compared it on a test set of 1,890 images with two commercial screening models (anonymized as S and Y). They report higher sensitivity for three diseases and higher specificity than the commercial models, and higher net benefit in a decision curve analysis on a hypothetical cohort of 100,000 people built from Chinese prevalence figures. In a separate experiment, models for diabetic retinopathy trained on public datasets (MESSIDOR-2, APTOS-2019 and IDRiD) lost less performance on the programme’s images when built on RETFound than when built on ResNet50 or EfficientNetB3 pretrained on ImageNet-21k. The authors call the design a cross-sectional study, on images already collected by the programme: development and test images were drawn from the same programme’s dataset, the test set contained more disease than the screened population, and the authors note that the model was not implemented in the field. In our reading, the comparison also sets a model adapted to the target images against products trained elsewhere, so it measures local adaptation as much as the foundation model.

Prospective silent deployment. Campanella et al. (2025) evaluated EAGLE retrospectively on 1,742 internal slides and on external cohorts from several institutions (AUC 0.847 internal and 0.870 over the 1,484 external slides pooled, which the abstract calls mean AUCs; the discussion gives a mean of 0.861 across external cohorts), and then in a prospective silent trial: from May to August 2024 the model ran in real time on newly scanned primary lung adenocarcinoma samples, with the trial’s components running in parallel with the clinical workflow. On 197 primary samples it reached an AUC of 0.890 against the next-generation sequencing assay MSK-IMPACT as the reference. With thresholds fixed beforehand on a separate pretrial cohort, the authors estimated how many rapid PCR-based tests an AI-assisted workflow could avoid: at the least stringent of three operating points, 43% of rapid tests, with a negative predictive value of 0.963 and a positive predictive value of 0.984, which they judged non-inferior to the rapid test’s historical performance. They propose EAGLE as a screening step, not a replacement for sequencing, and state that a prospective interventional trial, in which predictions guide management, would be needed for definitive evidence of utility.

Workflow studies. MedSAM’s authors measured annotation time on an adrenal tumour CT dataset not used in training: two experts annotated tumours slice by slice, or drew sparse axis markers every 3 to 10 slices, let MedSAM segment from them and revised the results, and the second pipeline reduced annotation time by 82.37% and 82.95% for the two experts (Ma et al., 2024). This measures the effort of producing masks, not a clinical outcome.

Randomized trials. None of the studies read for this article is a randomized trial of a foundation-model-based tool. In our reading, the pattern resembles the one that Clinical Validation and Regulation of Medical Imaging AI documents for medical imaging AI in general, where most studies stop at retrospective evaluation, with the silent trial an intermediate step between retrospective validation and interventional use. Research models released as foundation models carry no regulatory status unless a maker obtains it for a specific device (Digital Pathology notes the same for pathology).

Example: A Frozen-Feature Baseline Against a Task-Specific Model

The structure of a foundation-model evaluation can be reproduced on synthetic images with known factors. The script below generates 32 × 32 images with three independent binary factors: the orientation of a faint background texture (horizontal or vertical streaks), the size of bright blobs (small or large) and their number (2–4 or 8–12), with random gain, offset and noise. Each factor defines one task, and the other two act as nuisances, so one encoder can be tested on three tasks. A small CNN encoder is pretrained without labels on 6,000 images by masked image modelling, in the spirit of the masked autoencoder but simplified for a CNN: half of the 4 × 4 patches are set to zero and a decoder reconstructs them. Three arms are then compared on 1,000 test images, at 2, 8, 32 and 128 labels per class drawn from a separate labelled pool:

• a linear probe (logistic regression) on the pretrained, frozen encoder’s features;
• the same probe on the same architecture with random, untrained weights, a control that separates the effect of pretraining from that of the architecture;
• the same architecture trained from scratch on the labelled images alone, the task-specific model.

Seeds 0 to 4 set the labelled draw and the random and scratch initializations; the script prints the mean and range over them. It uses PyTorch 2.14.1 with one CPU thread, NumPy 2.5.4, SciPy 1.18.1 and scikit-learn 1.9.1, and took about four to five minutes per run on a laptop CPU, most of it in pretraining.

import numpy as np
import torch
import torch.nn as nn
from scipy.ndimage import gaussian_filter
from sklearn.linear_model import LogisticRegression

torch.set_num_threads(1)
H = 32


def make_images(n, rng):
    """Synthetic images with three independent binary factors (balanced)."""
    f = {k: rng.permutation(np.arange(n) % 2) for k in ("size", "orient", "dense")}
    yy, xx = np.mgrid[0:H, 0:H]
    X = np.empty((n, H, H), np.float32)
    for i in range(n):
        sig = (0.5, 3.0) if f["orient"][i] == 0 else (3.0, 0.5)
        bg = gaussian_filter(rng.normal(size=(H, H)), sig)
        img = 0.15 * bg / bg.std()
        nb = rng.integers(2, 5) if f["dense"][i] == 0 else rng.integers(8, 13)
        for _ in range(nb):
            s = rng.uniform(1.2, 1.8) if f["size"][i] == 0 else rng.uniform(2.4, 3.0)
            cy, cx = rng.uniform(3, 29, size=2)
            img += rng.uniform(0.6, 1.0) * np.exp(-((yy - cy) ** 2 + (xx - cx) ** 2) / (2 * s * s))
        img = rng.uniform(0.8, 1.2) * img + rng.uniform(-0.1, 0.1)
        X[i] = img + rng.normal(0, 0.05, size=(H, H))
    return torch.from_numpy(X[:, None]), f


def encoder():
    return nn.Sequential(
        nn.Conv2d(1, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
        nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
        nn.Conv2d(32, 64, 3, padding=1), nn.ReLU())


def decoder():
    return nn.Sequential(
        nn.Upsample(scale_factor=2), nn.Conv2d(64, 32, 3, padding=1), nn.ReLU(),
        nn.Upsample(scale_factor=2), nn.Conv2d(32, 16, 3, padding=1), nn.ReLU(),
        nn.Conv2d(16, 1, 3, padding=1))


def features(enc, X):
    with torch.no_grad():
        return enc(X).mean(dim=(2, 3)).numpy()


def pretrain(Xu, seed=0, epochs=15):
    """Masked image modelling: hide half of the 4x4 patches, reconstruct them."""
    torch.manual_seed(seed)
    enc, dec = encoder(), decoder()
    opt = torch.optim.Adam(list(enc.parameters()) + list(dec.parameters()), lr=1e-3)
    g = torch.Generator().manual_seed(seed)
    losses = []
    for ep in range(epochs):
        perm, tot = torch.randperm(len(Xu), generator=g), 0.0
        for b in range(0, len(Xu), 128):
            x = Xu[perm[b:b + 128]]
            keep = (torch.rand(len(x), 1, 8, 8, generator=g) > 0.5).float()
            keep = keep.repeat_interleave(4, 2).repeat_interleave(4, 3)
            rec = dec(enc(x * keep))
            loss = (((rec - x) ** 2) * (1 - keep)).sum() / (1 - keep).sum()
            opt.zero_grad(); loss.backward(); opt.step()
            tot += loss.item() * len(x)
        losses.append(tot / len(Xu))
    return enc.eval(), losses


def probe(Ftr, ytr, Fte, yte):
    clf = LogisticRegression(C=1.0, max_iter=5000).fit(Ftr, ytr)
    return (clf.predict(Fte) == yte).mean()


def scratch(Xtr, ytr, Xte, yte, seed, steps=300):
    torch.manual_seed(seed)
    net = nn.Sequential(encoder(), nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(64, 2))
    opt = torch.optim.Adam(net.parameters(), lr=1e-3)
    g = torch.Generator().manual_seed(seed)
    y = torch.from_numpy(ytr).long()
    bs = min(32, len(ytr))
    for _ in range(steps):
        idx = torch.randperm(len(ytr), generator=g)[:bs]
        loss = nn.functional.cross_entropy(net(Xtr[idx]), y[idx])
        opt.zero_grad(); loss.backward(); opt.step()
    with torch.no_grad():
        return (net(Xte).argmax(1).numpy() == yte).mean()


Xu, _ = make_images(6000, np.random.default_rng(100))     # unlabelled pool
Xl, fl = make_images(2000, np.random.default_rng(101))    # labelled pool
Xt, ft = make_images(1000, np.random.default_rng(102))    # test set

enc, losses = pretrain(Xu)
print(f"pretraining masked-pixel MSE: epoch 1 {losses[0]:.4f}, epoch 15 {losses[-1]:.4f}")
Fu = features(enc, Xu)
mu, sd = Fu.mean(0), Fu.std(0) + 1e-8
Fl, Ft = (features(enc, Xl) - mu) / sd, (features(enc, Xt) - mu) / sd

rand_feats = {}                     # random-init encoders, frozen, one per seed
for s in range(5):
    torch.manual_seed(s)
    renc = encoder().eval()
    Fr = features(renc, Xu); m2, s2 = Fr.mean(0), Fr.std(0) + 1e-8
    rand_feats[s] = ((features(renc, Xl) - m2) / s2, (features(renc, Xt) - m2) / s2)

tasks = {"A size": "size", "B orientation": "orient", "C density": "dense"}
seeds, budgets = range(5), (2, 8, 32, 128)
fmt = lambda a: f"{np.mean(a):.3f} [{np.min(a):.3f}, {np.max(a):.3f}]"
print("task           K   pretrained probe       random-frozen probe    scratch CNN")
for name, key in tasks.items():
    yl, yt = fl[key], ft[key]
    for K in budgets:
        res = {"pre": [], "rnd": [], "scr": []}
        for s in seeds:
            rng = np.random.default_rng(s)
            idx = np.concatenate([rng.choice(np.flatnonzero(yl == c), K, replace=False) for c in (0, 1)])
            res["pre"].append(probe(Fl[idx], yl[idx], Ft, yt))
            Rl, Rt = rand_feats[s]
            res["rnd"].append(probe(Rl[idx], yl[idx], Rt, yt))
            res["scr"].append(scratch(Xl[idx], yl[idx], Xt, yt, s))
        print(f"{name:14s} {K:3d}  {fmt(res['pre'])}  {fmt(res['rnd'])}  {fmt(res['scr'])}")

pretraining masked-pixel MSE: epoch 1 0.0701, epoch 15 0.0354
task           K   pretrained probe       random-frozen probe    scratch CNN
A size           2  0.645 [0.467, 0.737]  0.634 [0.450, 0.748]  0.666 [0.387, 0.783]
A size           8  0.822 [0.710, 0.882]  0.778 [0.689, 0.849]  0.847 [0.675, 0.910]
A size          32  0.906 [0.869, 0.943]  0.855 [0.821, 0.885]  0.947 [0.871, 0.975]
A size         128  0.957 [0.948, 0.963]  0.933 [0.920, 0.945]  0.965 [0.937, 0.981]
B orientation    2  0.629 [0.502, 0.756]  0.642 [0.569, 0.680]  0.817 [0.609, 0.998]
B orientation    8  0.928 [0.898, 0.951]  0.906 [0.853, 0.934]  1.000 [0.998, 1.000]
B orientation   32  0.978 [0.972, 0.984]  0.980 [0.961, 0.994]  1.000 [1.000, 1.000]
B orientation  128  0.990 [0.988, 0.992]  0.994 [0.979, 0.999]  1.000 [1.000, 1.000]
C density        2  0.778 [0.677, 0.850]  0.771 [0.582, 0.856]  0.722 [0.658, 0.770]
C density        8  0.847 [0.780, 0.874]  0.815 [0.774, 0.847]  0.824 [0.807, 0.856]
C density       32  0.887 [0.879, 0.894]  0.867 [0.833, 0.894]  0.909 [0.889, 0.934]
C density      128  0.918 [0.914, 0.924]  0.909 [0.881, 0.932]  0.916 [0.899, 0.927]

The masked-pixel error halves during pretraining, so the encoder learns something about the images, but the table shows that learning a reconstruction objective is not the same as learning every task. On blob size, the pretrained features beat the random ones at 8 labels per class and above, so pretraining helped, yet the scratch CNN was as good or better at every budget. On density, the pretrained probe was the best arm at 2 and 8 labels per class, fell behind the scratch CNN at 32 and was level with it at 128. On texture orientation, the pretrained probe’s mean stayed within the random-weight control’s range at every budget, and the scratch CNN solved the task almost perfectly from 8 labels per class. In our reading, the reconstruction loss is dominated by the bright blobs, which carry most of the pixel variance, while the low-amplitude texture contributes little; a probe on such features inherits that emphasis.

Three points carry over to the published evaluations. First, the result depends on the comparison arm: against the random-weight control the pretrained encoder’s mean is higher at every budget on two of three tasks (on density by less than the spread over seeds), and against a model trained for the task it wins only on density with few labels, so a claim of label efficiency means little without saying what it is relative to. Second, one encoder ranks differently across tasks, which is why many-task suites are used and why their averages can hide a task on which the encoder adds nothing. Third, the ranges at 2 labels per class reach 0.25 to 0.4 in accuracy for several arms and tasks, larger than most differences between the arms’ means, which is why UNI’s authors repeat few-shot settings over many draws. The example is a toy: its tasks are easy, its encoder is tiny and its pretraining pool is homogeneous, so it shows the structure of the evaluation, not the size of the effects the papers report.

The script shown differs from the first run in one respect: the random-weight encoders’ features are computed once per seed for the whole labelled pool and test set, instead of again for every task and budget; this removed repeated work and left the printed output unchanged. A shorter pretraining run (half the unlabelled pool, ten epochs) was also tried to cut the run time and was not adopted, because it changes the encoder being evaluated. The plan written before the first run expected the pretrained probe to beat the scratch CNN at 2 and 8 labels per class on most tasks; the printed results do not bear this out, and no settings were changed in response.

Common Misconceptions

• “A medical foundation model can be used clinically out of the box.” The models are research encoders. A clinical tool is a task model adapted from one and validated for its use, as EAGLE was evaluated prospectively for one biomarker at one centre.
• “Pretraining on medical images always beats ImageNet.” RETFound’s authors found that ImageNet initialization followed by retinal pretraining worked best, better than retinal pretraining alone, and the supervised ImageNet-21k model ranked second on most of their ocular diagnosis datasets.
• “Zero-shot means no labels were used.” It means no labelled training examples for the task. Thresholds and hyperparameters may still be tuned on labelled validation data, as in CheXzero.
• “A higher average over many tasks means a better encoder for my task.” Averages hide per-task differences, and independent benchmarks have ranked models differently from their own papers.
• “Excluding the test images from pretraining removes contamination.” It removes direct overlap. Pretraining on the same institutions, scanners or public collections as the test data still makes the evaluation less external than it appears.
• “More pretraining data always helps.” UNI’s authors found that a model pretrained on a smaller subset of their data outperformed the full-data model on one task after 50,000 training iterations, with the full-data model improving with longer training, and Neidlinger et al. (2025) found mostly non-significant correlations between pretraining-set size and downstream performance.

Where It Is Used

Medical foundation models are used as feature extractors and starting points across medical image analysis. In digital pathology, frozen tile encoders such as UNI and CONCH feed the aggregators of whole-slide image classification for subtyping, grading and biomarker prediction. In ophthalmic image analysis, RETFound is fine-tuned for disease detection and prognosis. In radiology and chest X-ray analysis, image–text and image-only encoders are used for zero-shot findings classification and as backbones for medical image classification. In medical image segmentation, promptable models such as MedSAM are used for interactive and semi-automatic annotation. The general methods behind them are explained in self-supervised learning, contrastive learning, CLIP, DINO and SAM.