Learning Transferable Visual Models From Natural Language Supervision

“Learning Transferable Visual Models From Natural Language Supervision” by Alec Radford, Jong Wook Kim and ten OpenAI colleagues was posted to arXiv on 26 February 2021 and published at the International Conference on Machine Learning (ICML) in July 2021; Radford and Kim are marked as equal contributors. It trained image and text encoders to match web images with their accompanying text, named the method CLIP, and showed that the resulting models classify images from many datasets zero-shot, from class names alone. The model itself is covered in the CLIP article; this article covers the publication and what it measured.

The 16-page ICML paper condenses the 48-page arXiv paper (which has a single version), and its 25-page supplement holds most of the remaining analyses. Unmarked claims appear in the ICML text; figure and table numbers are the arXiv ones.

Problem

By 2020, language models such as GPT-3, pretrained on raw web text, transferred zero-shot to many tasks, while vision still pretrained on crowd-labeled datasets such as ImageNet. The authors asked whether web text could do the same for vision.

Earlier attempts lagged far behind. Visual N-Grams (Li et al., 2017) reached 11.5% zero-shot accuracy on ImageNet, counting all 1,000 classes although only 326 were in its n-gram dictionary. VirTex, ICMLM and ConVIRT (2020) learned visual features from captions, but on one to two hundred thousand images. The strongest weakly supervised systems, trained on Instagram hashtags or JFT-300M labels, used fixed sets of 1,000 or 18,291 classes, which the arXiv introduction argues limits their zero-shot use. The authors named scale as the crucial difference.

Contribution

In its ICML related-work section the paper calls learning vision from language an old idea, its own contribution being the study at scale. It contributed:

• WIT, a dataset of 400 million (image, text) pairs collected from public internet sources.
• An efficiency study that selected a contrastive objective over caption prediction.
• Eight models, five ResNets and three Vision Transformers, spanning almost two orders of magnitude of compute (arXiv text), with code and weights released.
• A broad evaluation: zero-shot transfer and linear probes on over 30 datasets against over 50 existing systems, plus studies of robustness, data overlap, human performance, social bias and surveillance uses.

Method

Data collection. WIT’s construction (500,000 queries, at most 20,000 pairs each) is summarized in CLIP; its total word count is similar to GPT-2’s WebText. The supplement and arXiv appendix test whether WIT itself mattered: trained on the 15-million-photo filtered YFCC100M or on an equal-sized WIT subset, a ResNet-50 scored similarly on average (65.5 against 66.6 linear probe, 29.6 against 30.0 zero-shot, YFCC first), though single datasets moved by over 10 points, YFCC favoring birds and flowers and WIT cars. The authors suspected that WIT’s main advantage was its size.

Choosing the objective. Training efficiency decided the method. Figure 2, in which caption prediction as in VirTex learned zero-shot ImageNet recognition three times more slowly than bag-of-words prediction and a contrastive objective gave a further fourfold gain, is described in CLIP, as is the final recipe, a simplified ConVIRT trained from scratch.

Evaluation protocols. The authors measured task learning with zero-shot transfer, where class names in prompts become a linear classifier, and representation learning with linear probes, logistic regression on frozen features. In the arXiv text they prefer probes to fine-tuning because fine-tuning can mask features that failed to generalize, and because 66 models on 27 datasets already meant 1,782 evaluations. Probes ran on the 12-dataset suite of Kornblith et al. (2019) and on a broader 27-dataset suite adding OCR, geo-localization, facial expression, action and traffic-sign tasks, including two datasets the authors built (supplement), Country211 and Rendered SST2.

Results

All figures are for ViT-L/14 at 336 pixels, the paper’s default “CLIP”, unless another model is named.

Zero-shot transfer. Against Visual N-Grams (Table 1), accuracy rose from 72.4% to 98.4% on aYahoo, 11.5% to 76.2% on ImageNet, and 23.0% to 58.5% on SUN. The comparison is not controlled; as a closer check, a CLIP ResNet-50 trained on YFCC100M matched Visual N-Grams’ ImageNet result within one V100 GPU day. Across 27 datasets, zero-shot CLIP beat logistic regression on ResNet-50 features on 16, gaining 14.5 points on Kinetics-700, which the authors attribute speculatively to language supervising verbs as well as nouns.

Zero-shot versus few-shot. On the 20 datasets with at least 16 examples per class, zero-shot CLIP matched a 4-shot linear probe on its own features and roughly matched the best 16-shot probe on any other model’s features (BiT-M ResNet-152x2). Text names the concept directly, the authors argue, whereas one example image fits many hypotheses. In the supplement and arXiv text, the labeled examples per class that a probe needs to match zero-shot ranged from fewer than one (Flowers102, EuroSAT) to 184, with a median of 5.4 and 16 on ImageNet. Zero-shot and fully supervised probe accuracy correlated at $r = 0.82$, but zero-shot was mostly 10 to 25 points lower. Only on STL10, CIFAR10, Food101, OxfordPets and Caltech101 did zero-shot come within 3 points.

Linear probes. CLIP ViTs were about three times more compute-efficient than CLIP ResNets. The best model beat the best existing model, a Noisy Student EfficientNet-L2, by 2.6 points on average on the 12-dataset suite and by 5 on the 27-dataset suite. In the supplement and arXiv text it won on 21 of the 27, most on OCR, geo-localization, cars and traffic signs, and lost most on ImageNet (3.0 points).

Robustness to natural distribution shift. The analysis uses the framework of Taori et al. (2020), who found that across 204 ImageNet models, accuracy on shifted test sets is roughly a linear function of ImageNet accuracy after a logit transform. Effective robustness is accuracy above that line. All zero-shot CLIP models lay well above it, shrinking the gap between ImageNet and shifted-set accuracy by up to 75%. The supplement and arXiv text then pull the effect apart:

Intervention on CLIP’s features (arXiv Figure 14, supplement Figure 18)	Effect
Logistic regression fitted on ImageNet	ImageNet +9.2 (to 85.4%), ImageNetV2 +5.8; ImageNet-R −4.7, ObjectNet −3.8, ImageNet Sketch −2.8, ImageNet-A −1.9; average over the shifts slightly lower
Zero-shot classifier built from each dataset’s own class names	Youtube-BB +26.9, ImageNet-Vid +8.3, ObjectNet +2.3; about 5 points more average effective robustness
Few-shot probes, 1 to 128 shots, then fully supervised	Effective robustness fades as training examples increase; 16-shot matches zero-shot on ImageNet but is less robust

The supplement’s Table 16 implies a 6.1-point ObjectNet drop (72.3% to 66.2%), not 3.8; in our reading, its zero-shot figure uses ObjectNet’s own class names (+2.3), whereas Figure 14 starts from ImageNet names.

The authors concluded that high effective robustness seems to come from minimizing distribution-specific training data, at a cost in dataset-specific accuracy, but did not rule out the dataset or language supervision as causes.

Comparison with humans (supplement and arXiv). Five people labeled the 3,669 Oxford-IIIT Pets test images among 37 breeds. Their average went from 53.7% zero-shot to 75.7% with one example per class, with almost no further gain from a second; zero-shot CLIP scored 93.5% (all as average per-class accuracy). The authors contrast this one-example gain with CLIP’s few-shot probes, which make no effective use of prior knowledge.

Data overlap. Of 35 evaluation datasets compared with WIT, nine had no detected overlap; the median was 2.2% and the mean 3.2%. Accuracy shifted by more than 0.1 points on only 7 datasets, 2 of them significant after Bonferroni correction, and at most by 0.6 points (Birdsnap). The supplement and arXiv text add the method (a purpose-built near-duplicate detector, testing zero-shot CLIP RN50x64) and that Country211 overlapped most (21.5%, being built from YFCC100M) yet gained only 0.2 points, its matching training texts often not mentioning the location.

Bias and Surveillance Analyses

In the ICML text, adding labels such as “criminal” and “animal” to FairFace’s race classes put 32.3% of images of people aged 0–20 into those categories; adding the class “child” cut this to 8.7%. The arXiv version instead reports age bins (Table 7; supplement Table 9), for example 30.3% falling to 2.3% for ages 0–2 and 29.5% to 14.7% for 10–19. The supplement and arXiv Section 7 add more:

• FairFace probes. With 4 non-human and 3 crime-related labels added to 14 race-and-gender classes, 4.9% of images landed in a non-human class: about 14% for the “Black” category, under 8% for every other. Crime-related labels took 16.5% of male and 9.8% of female images.
• Members of Congress. At a 0.5% label-probability threshold, labels such as “nanny” and “housekeeper” appeared for women and “prisoner” and “mobster” for men.
• Surveillance. On 515 CCTV frames, coarse scene classification reached 91.8% top-1 but 51.1% when a near-miss caption was added. Zero-shot identification on CelebA, also reported in the ICML text (ViT-L/14 in arXiv Table 8), reached 59.2% top-1 among 100 celebrity names and 43.3% among 1,000. The authors judge this not competitive with production systems but warn that such models lower the skill needed for niche surveillance.

The authors stress throughout that class design, not only training data, shapes these harms, since anyone can define CLIP’s classes.

Impact

In our reading, the paper set the pattern for later image–text models: a contrastive dual encoder trained on hundreds of millions of uncurated web pairs and evaluated zero-shot on broad suites. ALIGN (Jia et al., 2021), at the same conference, trained a similar dual encoder on over one billion noisy image alt-text pairs. The released encoders became building blocks for image–text retrieval, open-vocabulary recognition and vision–language models.

Limitations

The ICML paper lists four limitations: zero-shot CLIP is often only competitive with a linear probe on ResNet-50 features, and reaching the overall state of the art would take an estimated 1,000 times more compute; the authors repeatedly queried validation sets during development, which a true zero-shot setting would not allow; their 27-dataset suite is “undeniably co-adapted” with CLIP’s capabilities; and many tasks are hard to specify in text, yet CLIP loses accuracy when moving from zero-shot to few-shot probes.

The arXiv Section 6 adds poor generalization to truly out-of-distribution data such as handwritten MNIST, the restriction to choosing among given texts (the authors suggest a joint contrastive and captioning objective), and data hunger: shown at one image per second, its 12.8 billion training images would take 405 years.

What Came After

• Robustness. Fang et al. (2022) tested five possible causes of CLIP’s robustness, including dataset size, language supervision during training and at test time, and the contrastive loss. The more diverse training distribution was the main cause, the others contributing little or nothing; language supervision on the same images did not add robustness. Wortsman et al. (2022) noted that fine-tuning often reduces robustness, as in CLIP’s own experiments, and proposed WiSE-FT, which interpolates the zero-shot and fine-tuned weights, $\theta_\alpha = (1 - \alpha)\,\theta_{\text{zero-shot}} + \alpha\,\theta_{\text{fine-tuned}}$.
• From zero-shot to few-shot. CoOp (Zhou et al., 2022) replaced hand-written prompt words with learned context vectors while keeping CLIP frozen. In the abstract of its arXiv version (v6), one or two examples per class sufficed to beat hand-crafted prompts on 11 datasets, and with 16 the average gain was around 15%.
• Open reproduction. The paper released code and weights but not WIT. Cherti et al. (2023) trained CLIP models with OpenCLIP on the public LAION data and found that OpenAI’s and OpenCLIP’s models scaled differently, which they attribute to the training distribution. Later models are surveyed in vision–language pretraining.