CLIP

CLIP (Contrastive Language–Image Pre-training) is a pair of encoders, one for images and one for text, trained so that an image and the text that came with it map to nearby points in a shared embedding space. Because the text side can embed any phrase, CLIP can classify images into categories it was never explicitly trained on: embed “a photo of a dog.”, “a photo of a cat.” and so on, and pick the prompt closest to the image. Radford and eleven OpenAI colleagues introduced it in a February 2021 preprint, published at ICML 2021, with ResNet and Vision Transformer image encoders and a transformer text encoder trained on 400 million image–text pairs from the internet. Its objective is the image–text form of contrastive learning, and its embeddings are a coordinated representation in the sense of multimodal learning.

The 16-page ICML paper condenses the 48-page arXiv version, and its supplement holds most of the arXiv appendices. Table and figure numbers below are the arXiv ones; details in neither ICML part are marked “arXiv text”.

Problem Addressed

Image classification models predict a fixed set of categories, so each new concept needs new labelled data. Learning from the text that accompanies web images had been tried: Visual N-Grams (Li et al., 2017) classified ImageNet zero-shot at 11.5%, and VirTex, ICMLM and ConVIRT (2020) learned from captions with transformer or contrastive objectives, but on one to two hundred thousand images. CLIP’s authors saw scale as the missing ingredient; CLIP is, in their framing, a simplified ConVIRT trained from scratch on far more data.

Architecture

The two towers do not attend to each other. Each produces a feature vector that a learned linear projection maps into the shared space, followed by $\ell_2$ normalization, so the dot product of an image and a text embedding is their cosine similarity. Unlike SimCLR, CLIP uses no nonlinear projection head; the arXiv text reports no difference in training efficiency. The embedding has 512 dimensions for the ViT-B models, 768 for ViT-L/14 and 1,024 for RN50 (Tables 19–20).

Image Encoder

• ResNets. ResNet-50 and ResNet-101 with the ResNet-D changes and antialiased blur pooling, and with global average pooling replaced by attention pooling: one multi-head attention layer whose query comes from the average-pooled features. RN50x4, RN50x16 and RN50x64 use roughly 4, 16 and 64 times ResNet-50’s compute, spread equally over width, depth and input resolution (288, 384 and 448 pixels).
• Vision Transformers. ViT-B/32, ViT-B/16 and ViT-L/14, with an extra layer normalization before the transformer. ViT-L/14@336px is ViT-L/14 trained one more epoch at 336 pixels; it gives the paper’s headline numbers.

Text Encoder

A GPT-2-style transformer with masked self-attention, 12 layers, width 512 and 8 heads, on lower-cased byte-pair-encoded text. The top-layer activation at the end-of-sequence token, layer-normalized, is the text feature. The arXiv text adds 63M parameters, a vocabulary of 49,152 and a maximum length of 76 tokens; the hyperparameter table (Table 18, also in the supplement) gives the vocabulary as 49,408. For larger image encoders only the text encoder’s width grew (to 1,024 for RN50x64), because performance was less sensitive to it.

Training

Data: WIT

MS-COCO and Visual Genome have about 100,000 photos each, and YFCC100M shrank from 100 million to 15 million photos once filtered to English natural-language titles or descriptions. The authors therefore built WIT (WebImageText): 400 million image–text pairs from publicly available internet sources, found by searching for text containing one of 500,000 queries, with at most 20,000 pairs per query to roughly balance concepts. The queries were the words occurring at least 100 times in English Wikipedia, plus frequent bigrams, popular Wikipedia article titles and WordNet synsets. OpenAI has not released WIT (model card).

Objective

In the notation of the contrastive learning article’s SigLIP entry, let $\mathbf{a}_i$ and $\mathbf{b}_j$ be the normalized image and text embeddings of a batch of $N$ pairs. The logits are $s_{ij} = \mathbf{a}_i^\top \mathbf{b}_j / \tau$, with the temperature learned as $t = \log(1/\tau)$. (The multimodal learning article writes the same loss with $s_{ij}$ the cosine similarity and $\tau$ kept separate.) The loss averages a cross-entropy that finds each image’s caption among the $N$ captions with one that finds each caption’s image:

$$\mathcal{L} = \frac{1}{2N} \sum_{i=1}^{N} \left[ -\log \frac{e^{s_{ii}}}{\sum_{j=1}^{N} e^{s_{ij}}} \;-\; \log \frac{e^{s_{ii}}}{\sum_{j=1}^{N} e^{s_{ji}}} \right].$$

The $N$ diagonal entries of the similarity matrix are positives and the $N^2 - N$ others negatives; the paper traces this construction to Sohn’s N-pair loss and ConVIRT (the arXiv text also to InfoNCE). With $P^{\text{txt}}_{ij}$ the row softmax (image $i$ choosing caption $j$) and $P^{\text{img}}_{ij}$ the column softmax,

$$\frac{\partial \mathcal{L}}{\partial s_{ij}} = \frac{P^{\text{txt}}_{ij} + P^{\text{img}}_{ij} - 2\,\mathbb{1}[i = j]}{2N}, \qquad \frac{\partial \mathcal{L}}{\partial t} = \sum_{i,j} \frac{\partial \mathcal{L}}{\partial s_{ij}}\, s_{ij}.$$

By our reading, the second gradient is negative, so the logit scale $1/\tau$ grows, whenever positives score on balance above the softmax-weighted averages of their rows and columns; once the training pairs are separated, little holds the scale back.

Why a Contrastive Objective

The authors first trained an image CNN and a text transformer to generate captions, as in VirTex. In their Figure 2, a 63M-parameter transformer language model learned zero-shot ImageNet recognition three times more slowly than a baseline predicting a bag-of-words encoding of the text, and swapping that prediction for the contrastive objective gave a further fourfold gain in efficiency. Predicting the exact words is hard because the text around images varies so much (arXiv text); the contrastive task only asks which text as a whole goes with which image.

Procedure

All models were trained from scratch, with a random square crop as the only augmentation. RN50x64 took 18 days on 592 V100 GPUs and ViT-L/14 12 days on 256. The arXiv text adds the rest: Adam with decoupled weight decay and a cosine schedule, 32 epochs (12.8 billion images), batches of 32,768, similarity computation sharded across GPUs, and a temperature initialized to the equivalent of 0.07 and clipped so logits are scaled by at most 100, which was needed for stability. Table 18, also in the supplement, lists the epochs, batch size and cap.

Example: the CLIP loss and its gradient

The script implements the paper’s Figure 3 pseudocode for eight $16 \times 16$ synthetic shape images, each with an invented caption such as “a small disk on the left”. The “image encoder” is the raw pixels and the “text encoder” a bag-of-words count (the pseudocode allows a CBOW text encoder), so only the projections $W_i$, $W_t$ and the temperature are learned. It checks the hand-written gradients against finite differences, then takes 300 gradient steps; it runs in under a second with NumPy 2.3.

import numpy as np

rng = np.random.default_rng(0)
S = 16
yy, xx = np.mgrid[0:S, 0:S] - 7.5

def shape(kind, size, dx):
    x, y = xx - dx, yy
    if kind == "disk":
        return x**2 + y**2 < size**2
    if kind == "square":
        return (abs(x) < size) & (abs(y) < size)
    if kind == "bar":
        return (abs(x) < size + 2) & (abs(y) < 1.5)
    return (x**2 + y**2 < size**2) & (x**2 + y**2 > (size - 2) ** 2)   # ring

# N = 8 (image, caption) pairs; the captions use an invented 11-word vocabulary
pairs = [(k, sz, dx) for k in ("disk", "square", "bar", "ring") for sz, dx in ((3, -3), (6, 2))]
vocab = ["a", "small", "large", "disk", "square", "bar", "ring", "on", "the", "left", "right"]
captions = [f"a {'small' if sz == 3 else 'large'} {k} on the {'left' if dx < 0 else 'right'}"
            for k, sz, dx in pairs]
I_f = np.stack([shape(*p).ravel() + 0.1 * rng.standard_normal(S * S) for p in pairs])  # [n, 256]
T_f = np.array([[c.split().count(w) for w in vocab] for c in captions], float)       # [n, 11]
n, d_e = len(pairs), 16

def clip_loss(W_i, W_t, t):
    """Figure 3 of Radford et al. (2021) plus hand-written gradients."""
    X, Y = I_f @ W_i, T_f @ W_t                     # project to the joint space
    nx = np.linalg.norm(X, axis=1, keepdims=True)
    ny = np.linalg.norm(Y, axis=1, keepdims=True)
    I_e, T_e = X / nx, Y / ny                       # l2_normalize
    logits = I_e @ T_e.T * np.exp(t)                # [n, n] scaled cosine similarities
    def softmax(z, axis):
        z = np.exp(z - z.max(axis, keepdims=True))
        return z / z.sum(axis, keepdims=True)
    P_txt = softmax(logits, 1)                      # each image picks one of n captions
    P_img = softmax(logits, 0)                      # each caption picks one of n images
    diag = np.arange(n)
    loss_i = -np.log(P_img[diag, diag]).mean()      # cross_entropy_loss(logits, labels, axis=0)
    loss_t = -np.log(P_txt[diag, diag]).mean()      # cross_entropy_loss(logits, labels, axis=1)
    loss = (loss_i + loss_t) / 2
    G = ((P_txt - np.eye(n)) + (P_img - np.eye(n))) / (2 * n)   # d loss / d logits
    dI_e = np.exp(t) * G @ T_e
    dT_e = np.exp(t) * G.T @ I_e
    dX = (dI_e - (dI_e * I_e).sum(1, keepdims=True) * I_e) / nx  # back through normalization
    dY = (dT_e - (dT_e * T_e).sum(1, keepdims=True) * T_e) / ny
    return loss, I_f.T @ dX, T_f.T @ dY, (G * logits).sum(), P_txt

W_i = 0.1 * rng.standard_normal((S * S, d_e))
W_t = 0.1 * rng.standard_normal((len(vocab), d_e))
t = np.log(1 / 0.07)                                # initial temperature 0.07: logits scaled by 14.3
loss, dW_i, dW_t, dt, _ = clip_loss(W_i, W_t, t)
print(f"caption 0: '{captions[0]}'")
print(f"loss at initialization: {loss:.4f}   (chance: ln {n} = {np.log(n):.4f})")

def fd(f, A, eps=1e-6):
    out = np.zeros_like(A)
    for idx in np.ndindex(*A.shape):
        Ap, Am = A.copy(), A.copy()
        Ap[idx] += eps
        Am[idx] -= eps
        out[idx] = (f(Ap) - f(Am)) / (2 * eps)
    return out

err_i = abs(fd(lambda A: clip_loss(A, W_t, t)[0], W_i) - dW_i).max()
err_t = abs(fd(lambda A: clip_loss(W_i, A, t)[0], W_t) - dW_t).max()
err_s = abs(fd(lambda a: clip_loss(W_i, W_t, a[0])[0], np.array([t]))[0] - dt)
print(f"max |analytic - finite difference|: W_i {err_i:.1e}, W_t {err_t:.1e}, t {err_s:.1e}")

# 300 steps of plain gradient descent on W_i, W_t and t
for step in range(300):
    loss, dW_i, dW_t, dt, P = clip_loss(W_i, W_t, t)
    W_i -= 0.5 * dW_i
    W_t -= 0.5 * dW_t
    t = min(t - 0.5 * dt, np.log(100))              # CLIP clips the logit scale at 100
loss, *_, P = clip_loss(W_i, W_t, t)
print(f"after 300 steps: loss {loss:.4f}, logit scale exp(t) = {np.exp(t):.1f}, "
      f"images matched to their own caption: {(P.argmax(1) == np.arange(n)).sum()}/{n}")

caption 0: 'a small disk on the left'
loss at initialization: 3.6070   (chance: ln 8 = 2.0794)
max |analytic - finite difference|: W_i 7.8e-10, W_t 5.4e-10, t 1.2e-10
after 300 steps: loss 0.0002, logit scale exp(t) = 100.0, images matched to their own caption: 8/8

The initial loss exceeds $\ln 8$ because random similarities are already multiplied by $1/0.07 \approx 14.3$. Over seeds 0–9 all pairs were always matched and the scale ended between 76 and 100, at the cap in eight runs; it stalls when the loss nears zero, because its gradient then vanishes too.

Inference

For zero-shot classification, CLIP embeds one text per class and takes a softmax over the scaled cosine similarities between the image and the $K$ class texts. The arXiv text notes that this is logistic regression with normalized inputs and weights and no bias, whose weights the text encoder generates. The class embeddings are computed once and cached, so each image costs one image-encoder pass and $K$ dot products.

Prompts. Captions are rarely single words, and class names can be ambiguous (ImageNet has construction cranes and crane birds). The template “A photo of a {label}.” raised ImageNet accuracy by 1.3% over the bare label, and task-specific templates such as “A photo of a {label}, a type of pet.” helped on fine-grained datasets.

Prompt ensembling. Averaging the embeddings of many templates gives one classifier at no extra test-time cost. On ImageNet, 80 templates added 3.5% over the default prompt, and engineering plus ensembling almost 5% (arXiv text and ICML supplement).

The same embeddings support retrieval in both directions. Zero-shot, CLIP reached 88.0% text R@1 and 68.7% image R@1 on Flickr30k (Table 13).

Example: zero-shot classification and prompt ensembling

The second script uses synthetic embeddings instead of a trained model: 400 encoded test images of eight shape classes and the text embeddings of 80 templates per class. In our toy, each wording adds its own noise to the class concept, and each template adds an offset shared by all classes.

import numpy as np

def normalize(x):
    return x / np.linalg.norm(x, axis=-1, keepdims=True)

rng = np.random.default_rng(4)
classes = ["disk", "square", "triangle", "cross", "ring", "bar", "star", "arrow"]
K, d, M = len(classes), 64, 80                       # 8 classes, 64-D embeddings, 80 templates
shared = rng.standard_normal(d)                      # every class is "a shape": similar concepts
concept = normalize(shared + 0.5 * rng.standard_normal((K, d)))

# 400 test images, already encoded: the class concept plus image-specific variation
labels = rng.integers(0, K, 400)
img = normalize(concept[labels] + 0.12 * rng.standard_normal((400, d)))

# Text embedding of template m filled with class k ("a photo of a {label}." and so on):
# the concept, an offset shared by all classes of that template, and wording noise
offset = 0.5 * normalize(rng.standard_normal((M, 1, d)))
txt = normalize(concept + offset + 0.08 * rng.standard_normal((M, K, d)))   # [M, K, d]

def zero_shot(W):
    """Classifier weights W [K, d] from text; returns accuracy and logits."""
    logits = 100 * img @ normalize(W).T              # cosine similarity times the logit scale
    return (logits.argmax(1) == labels).mean(), logits

def top3(W, i):
    z = zero_shot(W)[1][i]
    p = np.exp(z - z.max())
    p /= p.sum()                                     # softmax over the 8 class prompts
    return ", ".join(f"{classes[k]} {p[k]:.2f}" for k in np.argsort(p)[::-1][:3])

single = [zero_shot(txt[m])[0] for m in range(M)]
i = np.flatnonzero(zero_shot(txt[0])[1].argmax(1) != labels)[0]   # a mistake of template 0
print(f"test image {i}, a {classes[labels[i]]}")
print(f"  template 0 alone: {top3(txt[0], i)}")
print(f"  80-template mean: {top3(txt.mean(0), i)}")
print(f"single templates: accuracy {min(single):.3f} to {max(single):.3f}, mean {np.mean(single):.3f}")
print("ensemble of the first k templates: " +
      ", ".join(f"k={k}: {zero_shot(txt[:k].mean(0))[0]:.3f}" for k in (1, 4, 16, 80)))

test image 4, a ring
  template 0 alone: square 0.96, ring 0.04, arrow 0.00
  80-template mean: ring 1.00, star 0.00, bar 0.00
single templates: accuracy 0.573 to 0.770, mean 0.673
ensemble of the first k templates: k=1: 0.665, k=4: 0.892, k=16: 0.955, k=80: 0.975

Averaging cancels the wording noise and the shared offset moves all classes together, so accuracy rises from 67% (single templates, on average) to 97.5%. Over seeds 0–19, single templates averaged 52–79% and the ensemble 92.5–99.0%. The gain dwarfs CLIP’s 3.5% because the toy’s noise is set high; the mechanism, averaging in embedding space, is the same.

Performance

Results reported by Radford et al. (2021) for ViT-L/14@336px unless stated:

• Zero-shot ImageNet: 76.2% top-1 and 95% top-5, matching the original supervised ResNet-50 without using ImageNet’s 1.28 million training images. Zero-shot RN50 reached 59.6% (Table 11).
• Against supervised features: zero-shot CLIP beat logistic regression on ResNet-50 features on 16 of 27 datasets, by up to 28.9 points (Stanford Cars), and lost by up to 37.1 (EuroSAT).
• Linear probes: logistic regression on CLIP features reached 85.4% on ImageNet and beat the Noisy Student EfficientNet-L2 on 21 of 27 datasets.
• Scaling: average zero-shot error fell log-log linearly over the 44-fold compute range of the five ResNets.

Natural distribution shift. Compared with a ResNet-101 of equal ImageNet accuracy, zero-shot CLIP closed up to 75% of the gap between ImageNet and shifted-set accuracy (Figure 13; Figure 7 in the ICML paper):

Dataset	ResNet-101	Zero-shot CLIP
ImageNet	76.2	76.2
ImageNetV2	64.3	70.1
ImageNet-R (renditions)	37.7	88.9
ObjectNet	32.6	72.3
ImageNet Sketch	25.2	60.2
ImageNet-A (adversarial)	2.7	77.1

Table 16 gives 77.2 for ImageNet-A. A classifier fitted to CLIP features on ImageNet gained 9.2 points there but slightly lowered the average on the shifted sets; the authors concluded that high effective robustness seems to come from minimizing distribution-specific training data.

Strengths

• Open vocabulary: classes are defined in text at test time, without retraining.
• Cheap supervision: web image–text pairs need no annotation, and performance kept improving with compute.
• Broad features: linear probes were strong far from ImageNet, on rendered-text OCR, geo-localization and video action recognition.

Limitations

The paper lists these itself:

• Specialized and abstract tasks. Zero-shot CLIP was weak on fine-grained classes (aircraft, flowers), counting (CLEVRCounts), satellite images, lymph-node tumours and the distance to the nearest car.
• True out-of-distribution data. In the arXiv text, zero-shot accuracy on handwritten MNIST was 88%, below logistic regression on raw pixels; WIT contained almost no MNIST-like images.
• Fixed answer set. CLIP chooses among given texts and cannot generate a description. Linear probes on few examples can score below zero-shot, whereas humans improve from one example.
• Compute and evaluation. About 1,000 times more compute would be needed to reach the overall state of the art, and the authors repeatedly queried full validation sets while developing a model meant for zero-shot use.
• Social bias. In a bias probe (arXiv text and supplement), with labels such as “animal”, “gorilla”, “thief” and “criminal” added to FairFace classes, 4.9% of images fell into a non-human class, about 14% for FairFace’s “Black” category. Adding the class “child” cut how often people under 20 landed in the denigrating classes (ICML version: from 32.3% to 8.7%), showing how class design shapes harm.

Historical Importance

CLIP showed in 2021 that natural-language supervision from hundreds of millions of web pairs yields image features that rival supervised ImageNet features and transfer zero-shot. ALIGN reached the same conclusion at the same conference, open reproductions followed, and CLIP’s released encoders became parts of many later systems.

Variants

• ALIGN (Jia et al., 2021) trains a dual encoder on 1.8 billion image–alt-text pairs, collected like Conceptual Captions but with most cleaning replaced by simple size and frequency filters. With an EfficientNet-L2 image encoder, a BERT-Large text encoder and batches of 16,384, it reached 76.4% zero-shot ImageNet top-1 using CLIP’s prompt ensembling.
• LAION-5B and OpenCLIP. LAION-5B (Schuhmann et al., 2022) holds 5.85 billion Common Crawl image–text pairs (2.32 billion English) kept when OpenAI’s CLIP ViT-B/32 gave English pairs a cosine similarity of at least 0.28 (0.26 from a multilingual CLIP for other languages). It was withdrawn in December 2023 after a Stanford Internet Observatory report found links to suspected child sexual abuse material, and re-released as Re-LAION-5B in August 2024 with 2,236 links removed (LAION, 2024). Cherti et al. (2023) fitted scaling laws for CLIP trained on LAION; their ViT-H/14 reached 78.0% zero-shot ImageNet. WIT-trained models scaled better on zero-shot classification and LAION-trained ones on retrieval, which the authors attribute to the data.
• SigLIP (Zhai et al., 2023) replaces the batch softmax with a sigmoid loss per image–text pair (see the contrastive learning article).

CLIP as a component:

• Text-to-image generation. DALL·E 2 (Ramesh et al., 2022) generates a CLIP image embedding from a caption and decodes an image from it; Stable Diffusion v1 conditions on a frozen CLIP ViT-L/14 text encoder (CompVis repository).
• Open-vocabulary detection and segmentation. OWL-ViT (Minderer et al., 2022) fine-tunes a contrastively pre-trained image–text model (its own or a public CLIP) for text-conditioned detection. LSeg (Li et al., 2022) aligns per-pixel embeddings with labels embedded by CLIP’s text encoder. SAM’s exploratory text prompts use CLIP’s text encoder.
• Vision–language models. LLaVA (Liu et al., 2023) feeds a language model the features of CLIP’s ViT-L/14 image encoder.

Implementations

OpenAI released code and weights at github.com/openai/CLIP in stages: RN50 and ViT-B/32 first, RN50x64 and ViT-L/14 in January 2022, and ViT-L/14@336px in April 2022 (model card). OpenCLIP (github.com/mlfoundations/open_clip) reimplements training and provides models trained on open data such as LAION-2B (Cherti et al., 2023).