Vision–Language Pretraining

Vision–language pretraining (VLP) trains a model on a large collection of images paired with text, such as captions or alt-text scraped from the web, before it is adapted to any particular task. The pretrained model is then transferred: fine-tuned for visual question answering, image–text retrieval or captioning, or used zero-shot. The idea follows BERT (Devlin et al., 2019), which pretrained a transformer on unlabelled text with masked language modeling and fine-tuned it per task. VLP adds an image, and with it objectives that tie words to what is visible. The contrastive model CLIP has its own article, and Multimodal Learning places VLP in the wider field.

Purpose

Annotated vision–language data is scarce: in UNITER’s split, COCO Captions has 533K captions on 106K images and Visual Genome 5.06M region descriptions on 101K images (Chen et al., 2020), and each downstream dataset is smaller still. VLP learns visual concepts, words and their correspondence once, from millions to billions of loosely paired images and texts, so that each task needs only brief fine-tuning or none.

When to Use It

Pretrained vision–language weights are the default starting point for tasks that relate images to text. Pretraining from scratch pays off only when no public checkpoint covers the domain (medical images with reports, satellite images with descriptions) and enough paired data exists. For image-only tasks, a VLP vision encoder is one option among self-supervised and supervised encoders.

How It Works

Data

The data has grown by more than four orders of magnitude while curation has decreased:

• Curated captions. COCO and Visual Genome (about 100K images each) have human-written text.
• Filtered alt-text. Conceptual Captions (CC3M; Sharma et al., 2018) kept only 0.2% of candidate pairs from about 1 billion English web pages after image, text and image–text filters, then replaced proper names with hypernyms, giving about 3.3M pairs. CC12M (Changpinyo et al., 2021) relaxed those filters to reach 12.4M pairs for pretraining rather than captioning.
• Web scale. Only 15M of YFCC100M’s roughly 100M Flickr photos (Thomee et al., 2016) had usable English text, so CLIP’s authors collected WIT, 400M pairs (Radford et al., 2021). ALIGN kept most of the noise: 1.8B image–alt-text pairs with only simple image and frequency-based text filters (Jia et al., 2021).
• CLIP-filtered open data. LAION-400M dropped Common Crawl pairs whose CLIP image and text embeddings had cosine similarity below 0.3, a threshold chosen by human inspection (Schuhmann et al., 2021). LAION-5B used CLIP ViT-B/32 models (OpenAI’s for English, a multilingual one otherwise) with thresholds of 0.28 and 0.26, keeping 5.85 billion pairs, 2.32 billion of them English (Schuhmann et al., 2022).

By our reading, filtering with a pretrained CLIP model makes later models inherit that model’s notion of a good caption, along with its blind spots.

Objectives

Write $x$ for an image and $c = (c_1, \dots, c_T)$ for its caption. Following the multimodal learning and contrastive learning articles, the unimodal encoders give normalized embeddings $u_i = f(x_i)$ and $w_j = g(c_j)$ with similarity $s_{ij} = \mathrm{sim}(u_i, w_j)$ and temperature $\tau$. A fusion encoder reads both modalities and returns a joint vector $h(x, c)$ at a [CLS] token.

Image–text contrastive (ITC). The symmetric InfoNCE loss over a batch of $N$ pairs,

$$\mathcal{L}_{\text{itc}} = -\frac{1}{2N} \sum_{i=1}^{N} \left[ \log \frac{e^{s_{ii}/\tau}}{\sum_{j} e^{s_{ij}/\tau}} + \log \frac{e^{s_{ii}/\tau}}{\sum_{j} e^{s_{ji}/\tau}} \right],$$

is the objective of dual encoders such as CLIP and ALIGN; the contrastive learning article derives its gradient and the role of $\tau$. ALBEF applies it to the unimodal encoders of a fusion model, with negatives from queues of 65,536 features produced by momentum encoders, as in MoCo (Li et al., 2021).

Image–text matching (ITM). A linear head on $h(x, c)$ predicts whether the pair matches,

$$\mathcal{L}_{\text{itm}} = -\mathbb{E}\big[\, y \log p + (1 - y) \log (1 - p) \,\big], \qquad p = \sigma\big(\theta^\top h(x, c)\big),$$

with $y = 1$ for true pairs (ALBEF writes it as a two-class softmax, which is equivalent: a softmax over two logits is the sigmoid of their difference). UNITER built negatives by swapping in a random image or caption (Chen et al., 2020), and such negatives are mostly easy. ALBEF instead mines in-batch hard negatives from the ITC similarities: for image $i$, one negative caption is drawn with probability

$$q_{ij} = \frac{e^{s_{ij}/\tau}}{\sum_{k \neq i} e^{s_{ik}/\tau}}, \qquad j \neq i,$$

and one negative image is drawn likewise for each caption. The official code zeroes the positive’s weight before sampling. The similarities already exist for ITC, so mining is nearly free.

Masked language modeling (MLM). As in BERT, 15% of caption tokens are masked and predicted, here from the remaining words and the image:

$$\mathcal{L}_{\text{mlm}} = -\mathbb{E} \log P\big(c_m \mid c_{\setminus m}, x\big).$$

UNITER masked one modality at a time (conditional masking), so that a masked word is never paired with a masked region it describes, and kept BERT’s 80/10/10 split between [MASK], random and unchanged tokens (Chen et al., 2020). ViLT masked whole words: if “giraffe” is split into three word pieces and only the middle one is masked, the text alone gives it away, and the image is ignored (Kim et al., 2021).

Masked region modeling (MRM). UNITER zeroed 15% of the region features $v_m$ and trained three variants: regressing the output at each masked region onto its detector feature, $\sum_m \lVert \hat{r}_\theta(v_m) - r(v_m) \rVert_2^2$ (MRFR); classifying the region with the detector’s top class as label (MRC); and matching the detector’s full class distribution $\tilde{p}(v_m)$ with the predicted one, $\sum_m D_{\mathrm{KL}}\big(\tilde{p}(v_m) \,\Vert\, \hat{p}_\theta(v_m)\big)$ (MRC-kl). The image-side analogue on pixels is the masked autoencoder.

Word–region alignment (WRA). UNITER also scored the alignment of contextualized word embeddings $e_t$ and region embeddings $v_k$ with an optimal-transport distance, $\min_{T \in \Pi(a, b)} \sum_{t,k} T_{tk}\,\big(1 - \cos(e_t, v_k)\big)$, where $\Pi(a, b)$ holds transport plans with given marginals, approximated with the IPOT algorithm; ViLT adapted it to patches as word–patch alignment.

Captioning and prefix language modeling. A decoder generates the caption token by token, $\mathcal{L}_{\text{lm}} = -\sum_t \log P(c_t \mid c_{<t}, x)$. SimVLM’s PrefixLM treats the image patches and a random-length beginning of the text as a bidirectionally attended prefix and predicts the rest autoregressively (Wang et al., 2022). Unlike ITC, ITM and MLM, generative objectives let the model write captions and open-ended answers.

How to Apply It

In practice one starts from a released checkpoint, picks the head that matches the task (similarity for retrieval, ITM score for re-ranking, a decoder for captioning, a classifier or decoder for VQA), and fine-tunes, often at higher resolution; ALBEF pretrains at $256 \times 256$ and fine-tunes at $384 \times 384$, interpolating the patch position embeddings (Li et al., 2021).

Example: ITC, ITM with hard negatives, and image-conditioned MLM

The images are $16 \times 16$ synthetic pictures of a small or large, red, green, blue or yellow circle, square or triangle, with invented captions such as “small yellow square” (24 combinations). As a stand-in for a pretrained dual encoder, captions become sums of random word vectors and images pass through a linear map fitted by ridge regression. The script computes ITC on a batch of eight pairs and samples one ALBEF-style negative caption per image; trains logistic-regression ITM heads on $u \odot w$ (standing in for the fused [CLS] vector) with random or mined negatives; and predicts a masked word from the other two, with and without the image embedding. The hand-written ITM and MLM gradients agree with finite differences to $10^{-9}$ (checked separately). It runs in about a second with NumPy 2.3.

import sys
import numpy as np

S, D, TAU = 16, 16, 0.1
SIZES = {"small": 3.0, "large": 6.0}
COLORS = {"red": (1, .2, .2), "green": (.2, 1, .2), "blue": (.2, .2, 1), "yellow": (1, 1, .2)}
SHAPES = ("circle", "square", "triangle")
VOCAB = list(SIZES) + list(COLORS) + list(SHAPES)       # 9 content words ("a" is dropped)
COMBOS = [(z, c, s) for z in SIZES for c in COLORS for s in SHAPES]   # 24 captions
yy, xx = np.mgrid[0:S, 0:S] + 0.5
seed = int(sys.argv[1]) if len(sys.argv) > 1 else 0
rng = np.random.default_rng(seed)

def draw(size, color, shape):
    """A 16x16 RGB image of a jittered, noisy colored shape on a dark background."""
    y, x = yy - 8 - rng.uniform(-1, 1), xx - 8 - rng.uniform(-1, 1)
    r = SIZES[size] * rng.uniform(0.9, 1.1)
    if shape == "circle":
        m = x**2 + y**2 < r**2
    elif shape == "square":
        m = (abs(x) < 0.85 * r) & (abs(y) < 0.85 * r)
    else:
        m = (y < 0.8 * r) & (y > -r + 2 * abs(x))
    img = 0.1 + m[..., None] * np.array(COLORS[color]) * rng.uniform(0.7, 1.0)
    return (img + 0.05 * rng.standard_normal((S, S, 3))).ravel()

def bag(words):
    v = np.zeros(len(VOCAB))
    for w in words:
        v[VOCAB.index(w)] += 1
    return v

def normalize(A):
    return A / np.linalg.norm(A, axis=-1, keepdims=True)

def softmax(Z):
    Z = np.exp(Z - Z.max(-1, keepdims=True))
    return Z / Z.sum(-1, keepdims=True)

# --- A stand-in for a pretrained dual encoder ---------------------------------
# Text tower: sum of fixed random word vectors. Image tower: a linear map from
# pixels, fitted by ridge regression so each image lands near its caption.
E = rng.standard_normal((len(VOCAB), D))
text_enc = lambda caps: normalize(np.array([bag(c) for c in caps]) @ E)
train = [COMBOS[k] for k in rng.integers(0, 24, 2000)]
X = np.array([draw(*c) for c in train])
Wimg = np.linalg.solve(X.T @ X + 1.0 * np.eye(X.shape[1]), X.T @ (np.array([bag(c) for c in train]) @ E))
img_enc = lambda X: normalize(X @ Wimg)

# --- ITC: symmetric InfoNCE on a batch of 8 pairs -----------------------------
def itc(U, W, tau=TAU):
    S_ = U @ W.T / tau                          # logits s_ij / tau
    loss = -0.5 * (np.log(np.diag(softmax(S_))).mean() + np.log(np.diag(softmax(S_.T))).mean())
    return loss, softmax(S_), softmax(S_.T)

batch = [COMBOS[k] for k in rng.choice(24, 8, replace=False)]
U, W = img_enc(np.array([draw(*c) for c in batch])), text_enc(batch)
loss, P_i2t, P_t2i = itc(U, W)
print(f"ITC loss on the batch: {loss:.3f}   (chance: ln 8 = {np.log(8):.3f})")

# --- ALBEF-style hard negatives: sample from the ITC softmax without the positive
def mine(P):
    Q = P.copy()
    np.fill_diagonal(Q, 0)
    Q /= Q.sum(1, keepdims=True)
    return np.array([rng.choice(len(Q), p=q) for q in Q]), Q

neg_t, Q = mine(P_i2t)
print("\nimage (true caption)        -> sampled negative caption     p    shared words")
for i, j in enumerate(neg_t):
    shared = len(set(batch[i]) & set(batch[j]))
    print(f"  {' '.join(batch[i]):24s} -> {' '.join(batch[j]):24s} {Q[i, j]:.2f}  {shared}")
share = np.array([[len(set(a) & set(b)) for b in batch] for a in batch])
print("negative probability mass on captions sharing 2 of 3 words: "
      f"{Q[share == 2].sum() / 8:.2f} (uniform sampling: {(share == 2).sum() / 56:.2f})")

# --- ITM: a binary head on a fused feature, trained with random or hard negatives
fuse = lambda U, W: np.hstack([U * W, np.ones((len(U), 1))])   # stand-in for the fused [CLS]

def itm_loss(theta, F, y, grad=False):
    p = 1 / (1 + np.exp(-F @ theta))
    L = -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))
    return (L, F.T @ (p - y) / len(y)) if grad else L

def make_pairs(n, how):
    """n positives and n negatives; negatives random or mined from ITC similarities."""
    caps = [COMBOS[k] for k in rng.integers(0, 24, n)]
    U = img_enc(np.array([draw(*c) for c in caps]))
    Wall = text_enc(COMBOS)
    pos = [COMBOS.index(c) for c in caps]
    if how == "random":
        neg = [(p + rng.integers(1, 24)) % 24 for p in pos]
    elif how == "hard":                          # ALBEF: p(j) ~ exp(s_ij / tau), j != positive
        P = softmax(U @ Wall.T / TAU)
        P[np.arange(n), pos] = 0
        neg = [rng.choice(24, p=q / q.sum()) for q in P]
    else:                                        # test set: exactly one word changed
        neg = []
        for c in caps:
            k = rng.integers(3)
            pool = [list(SIZES), list(COLORS), list(SHAPES)][k]
            new = list(c)
            new[k] = rng.choice([w for w in pool if w != c[k]])
            neg.append(COMBOS.index(tuple(new)))
    F = np.vstack([fuse(U, Wall[pos]), fuse(U, Wall[neg])])
    return F, np.r_[np.ones(n), np.zeros(n)]

def train_itm(how, steps=500, lr=2.0):
    theta = np.zeros(D + 1)
    F, y = make_pairs(1000, how)
    for _ in range(steps):
        theta -= lr * itm_loss(theta, F, y, grad=True)[1]
    return theta

tests = {name: make_pairs(500, name) for name in ("random", "hard", "one-word")}
heads = {how: train_itm(how) for how in ("random", "hard")}
print("\nITM loss of the random-trained head on test pairs with",
      ", ".join(f"{name.replace('hard', 'mined')} negatives {itm_loss(heads['random'], *tests[name]):.3f}"
                for name in tests))
print("ITM accuracy       random negatives  mined negatives  one-word-changed negatives")
for how, theta in heads.items():
    acc = [((F @ theta > 0) == y).mean() for F, y in tests.values()]
    print(f"  {how + '-trained':15s}  {acc[0]:.2f}              {acc[1]:.2f}             {acc[2]:.2f}")

# --- MLM: predict a masked word from the other words, with or without the image
def mlm_data(n):
    caps = [COMBOS[k] for k in rng.integers(0, 24, n)]
    U = img_enc(np.array([draw(*c) for c in caps]))
    k = rng.integers(3, size=n)                  # which word to mask
    ctx = np.array([bag([w for j, w in enumerate(c) if j != kk]) for c, kk in zip(caps, k)])
    tgt = np.array([VOCAB.index(c[kk]) for c, kk in zip(caps, k)])
    return ctx, U, tgt, k

def mlm_loss(V, F, tgt, grad=False):
    P = softmax(F @ V)
    L = -np.log(P[np.arange(len(tgt)), tgt]).mean()
    if not grad:
        return L
    P[np.arange(len(tgt)), tgt] -= 1
    return L, F.T @ P / len(tgt)

def fit_mlm(F, tgt, steps=1500, lr=2.0):
    V = np.zeros((F.shape[1], len(VOCAB)))
    for _ in range(steps):
        V -= lr * mlm_loss(V, F, tgt, grad=True)[1]
    return V

ctx, Utr, tgt, _ = mlm_data(2000)
ctx_te, Ute, tgt_te, k_te = mlm_data(1000)
one, one_te = np.ones((2000, 1)), np.ones((1000, 1))
print("\nmasked-word prediction (test cross-entropy by masked word type)")
for name, F, Fte in [("text only", np.hstack([ctx, one]), np.hstack([ctx_te, one_te])),
                     ("text + image", np.hstack([ctx, Utr, one]), np.hstack([ctx_te, Ute, one_te]))]:
    V = fit_mlm(F, tgt)
    per = [mlm_loss(V, Fte[k_te == k], tgt_te[k_te == k]) for k in range(3)]
    acc = ((Fte @ V).argmax(1) == tgt_te).mean()
    print(f"  {name:12s}  size {per[0]:.3f}  color {per[1]:.3f}  shape {per[2]:.3f}  accuracy {acc:.2f}")
print(f"  (uniform guess within the type: ln 2 = {np.log(2):.3f}, ln 4 = {np.log(4):.3f}, ln 3 = {np.log(3):.3f})")

ITC loss on the batch: 0.323   (chance: ln 8 = 2.079)

image (true caption)        -> sampled negative caption     p    shared words
  small yellow square      -> large yellow square      0.47  2
  large green circle       -> small green triangle     0.88  1
  small green triangle     -> small yellow circle      0.14  1
  large red square         -> large yellow square      0.86  2
  large yellow square      -> large red square         0.82  2
  small blue circle        -> small yellow square      0.55  1
  small yellow triangle    -> small yellow circle      0.68  2
  small yellow circle      -> small blue circle        0.11  2
negative probability mass on captions sharing 2 of 3 words: 0.65 (uniform sampling: 0.25)

ITM loss of the random-trained head on test pairs with random negatives 0.289, mined negatives 0.541, one-word negatives 0.484
ITM accuracy       random negatives  mined negatives  one-word-changed negatives
  random-trained   0.94              0.67             0.74
  hard-trained     0.97              0.91             0.95

masked-word prediction (test cross-entropy by masked word type)
  text only     size 0.703  color 1.395  shape 1.121  accuracy 0.35
  text + image  size 0.013  color 0.050  shape 0.089  accuracy 0.99
  (uniform guess within the type: ln 2 = 0.693, ln 4 = 1.386, ln 3 = 1.099)

The mining concentrates on near misses: 65% of the negative probability falls on captions that differ from the true one in a single word, against 25% under uniform sampling, so “large red square” draws “large yellow square”. The ITM head trained on random negatives finds them easy (loss 0.289) but has higher losses on mined and one-word negatives (0.541 and 0.484) and misclassifies 26% of the one-word test pairs; trained on mined negatives, it reaches 95% on them. For MLM, the text-only predictor can only guess within the masked word’s type (losses near $\ln 2$, $\ln 4$, $\ln 3$); with the image it predicts 99% of masked words. Over seeds 0–9: mined negatives took 57–85% of the probability (uniform 21–36%); one-word accuracy was 74–81% for the random-trained head and 88–95% for the hard-trained one; MLM accuracy was 33–37% without the image and 98–99% with it.

Variants

Region-feature fusion encoders (2019–2020). Bottom-up features (Anderson et al., 2018) came from a Faster R-CNN with a ResNet-101 backbone trained on Visual Genome with 1,600 object and 400 attribute classes: up to 100 confident regions per image, each a 2,048-dimensional mean-pooled feature. ViLBERT fed such regions and words into two transformer streams joined by co-attention, pretrained with masked multimodal modeling and alignment prediction on Conceptual Captions (Lu et al., 2019). UNITER used one stream over regions and words with the four objectives above, on 5.6M COCO and Visual Genome pairs plus about 4M from CC and SBU Captions; its ablation on the COCO and Visual Genome data found MLM + ITM + MRC-kl + MRFR + WRA best (Chen et al., 2020). Oscar added detected object tags as words that anchor the alignment, with a masked token loss and a contrastive loss on tag-polluted inputs, over 6.5M pairs (Li et al., 2020).

Detector-free fusion (2021). ViLT replaced the detector with the linear patch embedding of the Vision Transformer, initialized from ViT-B/32 and trained with ITM, word–patch alignment and whole-word MLM. Its authors timed the visual embedding and transformer at about 15 ms per input against about 900 ms for region-feature models such as UNITER-Base, most of it in the detector; ViLT-B/32 scored 74.57% on NLVR2 test-P against 75.80% for UNITER-Base, and 70.33% on VQAv2 test-dev against 72.70% (Kim et al., 2021).

Dual encoders (2021). CLIP and ALIGN train only ITC: retrieval over large collections is cheap, but there is no fusion layer for tasks such as VQA.

Align before fuse (2021). ALBEF combines the two: a ViT-B/16 image encoder and a 6-layer text encoder trained with ITC, then a 6-layer fusion encoder with cross-attention trained with ITM and MLM. Momentum distillation adds pseudo-targets from an exponential-moving-average copy of the model, weighted 0.4 against the one-hot targets, because with noisy captions some “negatives” are plausible matches and some unannotated words are correct (Li et al., 2021).

Unified understanding and generation (2022). BLIP shares one text transformer among a unimodal encoder (ITC), an image-grounded encoder (ITM with ALBEF’s hard negatives) and an image-grounded decoder (LM), with separate self-attention layers for encoding and decoding. Its CapFilt step bootstraps the data: a captioner, fine-tuned on COCO, writes a synthetic caption for each web image, and a filter, fine-tuned with ITC and ITM, removes web and synthetic captions it judges unmatched (Li et al., 2022). SimVLM pretrained with PrefixLM alone, on ALIGN’s pairs mixed with text-only documents (Wang et al., 2022). CoCa adds contrastive learning to a captioner: the first half of the decoder runs without cross-attention to give a text embedding for the contrastive loss, and the second half attends to the image for captioning, with $\mathcal{L} = \lambda_{\text{con}} \mathcal{L}_{\text{con}} + \lambda_{\text{cap}} \mathcal{L}_{\text{cap}}$ and $\lambda_{\text{cap}} = 2$, $\lambda_{\text{con}} = 1$; it treated JFT labels as text alongside ALIGN’s alt-text (Yu et al., 2022, arXiv version).

Frozen encoders and language models (2023). BLIP-2 keeps a pretrained image encoder and a large language model frozen and trains a 188M-parameter Querying Transformer (Q-Former) between them. Its 32 learned 768-dimensional queries summarize image features of size $257 \times 1024$ (ViT-L/14). Stage one trains the Q-Former on the frozen image encoder with ITC (taking the most similar query, with in-batch negatives), image-grounded text generation and ITM, using a different attention mask for each; stage two projects the queries into the language model’s input as soft visual prompts and trains with language modeling (Li et al., 2023).

Sigmoid loss. SigLIP replaces ITC’s batch softmax with an independent binary loss for every image–text pair (Zhai et al., 2023); see the contrastive learning article.

Model	Year	Architecture	Objectives	Pretraining data
ViLBERT	2019	two-stream, region features, co-attention	masked multimodal modeling, alignment prediction	CC (about 3.3M)
UNITER	2020	single-stream, region features	MLM, MRM, ITM, WRA	5.6M pairs (COCO, VG) + about 4M (CC, SBU)
ViLT	2021	single-stream, linear patch embedding	ITM, word–patch alignment, MLM	about 4M images (COCO, VG, CC, SBU)
CLIP	2021	dual encoder	ITC	400M pairs (WIT)
ALIGN	2021	dual encoder	ITC	1.8B pairs
ALBEF	2021	unimodal encoders + fusion encoder	ITC, ITM (hard negatives), MLM, momentum distillation	4M or 14M images
SimVLM	2022	encoder–decoder with convolutional patch stem	PrefixLM	ALIGN pairs + C4 text
BLIP	2022	shared encoder/decoder (MED)	ITC, ITM, LM; CapFilt	14M or 129M images
CoCa	2022	image encoder + unimodal and multimodal text decoder	contrastive + captioning	JFT-3B + ALIGN
BLIP-2	2023	frozen image encoder, Q-Former, frozen LLM	ITC, ITM, image-grounded generation; then LM	129M images

Best Practices

• Combine an alignment objective with a fusion objective. In ALBEF’s ablation (4M images), adding ITC to MLM + ITM raised the mean of R@1, R@5 and R@10 for Flickr30K text retrieval from 93.96 to 96.55 and VQAv2 test-dev accuracy from 71.40 to 73.29; hard negatives for ITM added a further 0.46 and 0.52 points (Li et al., 2021).
• Treat caption quality as a training variable. BLIP’s captioner and filter together raised fine-tuned COCO text-retrieval R@1 from 78.4 to 80.6 on its 14M-image set (Li et al., 2022).

Pitfalls

• Benchmark contamination. Web crawls contain copies of benchmark images. CLIP’s authors measured a median overlap of 2.2% across 35 datasets (Radford et al., 2021); ALIGN removed near-duplicates of test images from ImageNet, Flickr30K and COCO (Jia et al., 2021), and corpora built from COCO and Visual Genome must exclude downstream validation and test images, as UNITER did (Chen et al., 2020).
• Noisy and biased text. Alt-text often fails to describe its image (ALIGN’s examples include file names), and web data carries social biases into the model.
• Detector bottlenecks. Region-feature models inherit the detector’s fixed vocabulary (1,600 object classes for the bottom-up detector) and its running time (Kim et al., 2021).
• Compute. CLIP’s RN50x64 trained for 18 days on 592 V100 GPUs (Radford et al., 2021); BLIP-2 freezes its encoders partly to cut this cost.
• Confounded comparisons. In our reading, published gains mix objectives, architectures and data scales, so a better score does not isolate the objective.

Evidence and Impact

VLP models are evaluated zero-shot (image–text retrieval on Flickr30K and COCO, ImageNet classification by prompts) and after fine-tuning (VQAv2, NLVR2, COCO captioning). Some reported results: ALBEF with 14M images reached 95.9% text-retrieval R@1 on Flickr30K after fine-tuning, against 95.3% for ALIGN, listed with 1.2B images (Li et al., 2021); CoCa reported 86.3% zero-shot ImageNet top-1 (Yu et al., 2022, arXiv version); and BLIP-2 with ViT-g and FlanT5-XXL reached 65.0% zero-shot on VQAv2 test-dev, against 56.3% for Flamingo80B, with 54 times fewer trainable parameters (Li et al., 2023). The image–text retrieval article covers the retrieval benchmarks.