BLIP

BLIP (Bootstrapping Language-Image Pre-training) is a family of vision–language pretraining models from Salesforce Research. The original BLIP, by Junnan Li, Dongxu Li, Caiming Xiong and Steven Hoi (ICML 2022), is a transformer that serves both understanding tasks, such as image–text retrieval and visual question answering, and generation tasks such as image captioning. It made two contributions: a multimodal mixture of encoder–decoder (MED), one text transformer that can run in three modes, and CapFilt, which cleans the web captions it trains on with a learned captioner and filter. BLIP-2 (Li, Li, Savarese and Hoi, ICML 2023) keeps a pretrained image encoder and a large language model (LLM) frozen and trains only a small Querying Transformer (Q-Former) between them, which makes it an early vision–language model of the connector type. InstructBLIP (Dai et al., NeurIPS 2023) instruction-tunes the BLIP-2 Q-Former. The Vision–Language Pretraining article places BLIP among other pretraining methods; this article covers the models themselves.

Problem Addressed

In 2021, pretrained vision–language models came in two kinds. Encoder-based models, such as the dual encoder CLIP and the fusion model ALBEF, scored well on retrieval but could not easily generate text; encoder–decoder models generated captions but, according to the BLIP authors, had not been used successfully for retrieval. Both kinds trained mostly on image–alt-text pairs crawled from the web, and gains had come from scaling that data up. The BLIP authors argued that alt-text often does not describe its image, so that the noise in web text limits what scaling can buy (Li et al., 2022, §1).

By 2023 the cost had become the problem. End-to-end pretraining of large models was expensive, and it could not reuse the LLMs that the language community was releasing. Earlier attempts to attach a frozen LLM, Frozen and Flamingo, trained the visual side only with an image-to-text generation loss, which the BLIP-2 authors argued is not enough to bridge the gap between image features and the LLM’s input space (Li et al., 2023, §1–2).

Architecture

BLIP has a Vision Transformer image encoder and a text transformer initialized from BERT-base. The text transformer runs in three modes that share most of their weights. BLIP-2 replaces the trainable image encoder with a frozen one, adds a frozen LLM, and puts the Q-Former, a descendant of the MED, between them.

Image Encoder

BLIP uses ViT-B/16 or ViT-L/16, initialized from ImageNet-pretrained weights, which turns the image into patch embeddings plus a [CLS] embedding for the whole image. The authors chose a ViT over the object-detector features of earlier models, such as UNITER, because it is cheaper to compute (Li et al., 2022, §3.1, §4.1). The released code loads DeiT-B weights for the base model (models/blip_pretrain.py).

Multimodal Mixture of Encoder–Decoder

The MED operates in one of three modes (Li et al., 2022, §3.1):

• Unimodal encoder. The text transformer reads the caption alone, like BERT, with a [CLS] token at the start; the image encoder reads the image alone. Their [CLS] outputs are projected to a shared embedding space.
• Image-grounded text encoder. Each transformer block gains a cross-attention layer between its self-attention layer and its feed-forward network, attending to the image patch embeddings. An [Encode] token replaces [CLS], and its output is the joint representation of the image–text pair.
• Image-grounded text decoder. The bidirectional self-attention layers are replaced by causal ones, so each token sees only the tokens before it; a [Decode] token starts the sequence and an end-of-sequence token ends it.

The encoder and decoder share all parameters except the self-attention layers: the embeddings, cross-attention layers and feed-forward networks are common. The authors reason that encoding and decoding differ mainly in how tokens attend to each other. In their ablation on the 14M-image set, sharing everything but self-attention (252M parameters) gave fine-tuned COCO text-retrieval R@1 78.4 and COCO captioning CIDEr 127.8; sharing everything (224M) gave 77.3 and 125.9, and sharing nothing (361M) 78.3 and 127.4 (Table 3). The released code builds the decoder as a second BERT model and ties its weights to the encoder’s, skipping the self-attention modules (models/blip_pretrain.py, tie_encoder_decoder_weights).

Each training pair costs one forward pass of the image encoder and three of the text transformer, one per mode.

Querying Transformer (BLIP-2)

The Q-Former extracts a fixed number of visual features from a frozen image encoder, whatever the image resolution (Li et al., 2023, §3.1). It has two transformer submodules that share their self-attention layers:

• an image transformer, whose inputs are 32 learned query embeddings of dimension 768. The queries attend to each other through self-attention and to the frozen image features through cross-attention layers, inserted in every other block;
• a text transformer that works as a text encoder or a text decoder.

The queries can also attend to the text through the shared self-attention layers, and a different attention mask for each training objective controls whether they do. The Q-Former is initialized from BERT-base except for the randomly initialized cross-attention layers, and has 188M parameters, counting the queries as parameters. Its output $Z$ ($32 \times 768$) is much smaller than the frozen features ($257 \times 1024$ for ViT-L/14), and the authors describe it as an information bottleneck that passes on the visual information most relevant to text. In code, num_query_token=32 and cross_attention_freq=2 (LAVIS, blip2_qformer.py).

The frozen image encoder is either CLIP’s ViT-L/14 or EVA-CLIP’s ViT-g/14, with the last layer removed; the second-to-last layer’s features worked slightly better. The frozen LLM is from the OPT family (decoder-only) or the instruction-tuned FlanT5 family (encoder–decoder). A fully connected layer maps the 32 query outputs to the LLM’s embedding dimension, and the results are prepended to the text embeddings as soft visual prompts (§3.3–3.4). This is the “query-based resampler” connector of the vision–language model article, which compares it with LLaVA’s per-patch projector and Flamingo’s Perceiver Resampler.

Training

BLIP Objectives

BLIP optimizes three objectives jointly, one per MED mode, and the released code adds them without weights, $\mathcal{L} = \mathcal{L}_{\text{itc}} + \mathcal{L}_{\text{itm}} + \mathcal{L}_{\text{lm}}$ (pretrain.py). The first two come from ALBEF (Li et al., 2021), whose formulation is followed here; the Vision–Language Pretraining article derives the basic contrastive and matching losses.

Image–text contrastive loss (ITC) trains the unimodal encoders so that matching pairs have similar embeddings. As in ALBEF, a momentum encoder, an exponential moving average $\theta' \leftarrow m\,\theta' + (1 - m)\,\theta$ of the unimodal encoders with $m = 0.995$, supplies features for queues of recent images and texts, which serve as extra negatives. With $g_v$ and $g_w$ the projections of the image and text [CLS] outputs and primes marking momentum features, the image-to-text similarity distribution over the $M$ texts in the batch and queue is

$$p^{\text{i2t}}_m(I) = \frac{\exp\big(s(I, T_m)/\tau\big)}{\sum_{m'=1}^{M} \exp\big(s(I, T_{m'})/\tau\big)}, \qquad s(I, T) = g_v(v_{\text{cls}})^\top g'_w(w'_{\text{cls}}),$$

with a learnable temperature $\tau$, and $p^{\text{t2i}}$ likewise. BLIP adds ALBEF’s momentum distillation: because some web “negatives” in fact match the image, the one-hot targets $y$ are softened with pseudo-targets $q$, computed in the same way from momentum features on both sides:

$$\mathcal{L}_{\text{itc}} = (1 - \alpha)\,\tfrac{1}{2}\,\mathbb{E}\big[H(y^{\text{i2t}}, p^{\text{i2t}}) + H(y^{\text{t2i}}, p^{\text{t2i}})\big] + \tfrac{\alpha}{2}\,\mathbb{E}\big[\mathrm{KL}(q^{\text{i2t}} \,\Vert\, p^{\text{i2t}}) + \mathrm{KL}(q^{\text{t2i}} \,\Vert\, p^{\text{t2i}})\big],$$

where $H$ is cross-entropy (ALBEF Eq. 6, with $\alpha = 0.4$). BLIP’s code instead takes the cross-entropy against the mixed target $\alpha q + (1 - \alpha) y$; by our arithmetic this differs from the formula above only by a multiple of the entropy of $q$, which has no gradient, so the two give the same updates. The released configuration uses $\alpha = 0.4$, ramped up from 0 over the first two epochs, and a queue of 57,600 features, where ALBEF used 65,536 and a one-epoch ramp (configs/pretrain.yaml; pretrain.py; Li et al., 2021, §3.5).

Image–text matching loss (ITM) trains the image-grounded encoder with a linear head on the [Encode] output to classify pairs as matched or unmatched, a two-class cross-entropy. The negatives are ALBEF’s in-batch hard negatives: for each image one caption from the batch is sampled, captions with higher contrastive similarity being more likely, and for each caption one image (Li et al., 2022, §3.2); the Vision–Language Pretraining article gives the sampling distribution and a worked example. In the code, each batch of $B$ pairs yields $B$ positives and $2B$ negatives for the ITM head, sampled from the softmax over current-batch similarities only, not the queue (models/blip_pretrain.py).

Language modeling loss (LM) trains the decoder to generate the caption autoregressively,

$$\mathcal{L}_{\text{lm}} = -\sum_{t=1}^{T} \log P_\theta(c_t \mid c_{<t}, I),$$

with label smoothing of 0.1 (the target puts $1 - 0.1$ on the true token and spreads $0.1$ uniformly over the vocabulary, in PyTorch’s definition used by the code, models/med.py). The authors chose LM over the masked language modeling of earlier models because it lets the pretrained model write coherent captions (§3.2).

BLIP Data and Procedure

The 14M-image pretraining set is ALBEF’s: two human-annotated datasets, COCO (113K images, 567K texts) and Visual Genome (100K images, 821K texts), and three web datasets, SBU Captions (860K), Conceptual Captions (3M) and Conceptual 12M (10M). A larger set adds 115M images from LAION-400M, kept only when the shorter side exceeds 256 pixels, for 129M images; a fifth of LAION is used in each epoch (Li et al., 2022, §4.1, Table 15). Training runs for 20 epochs with batch 2,880 (ViT-B) or 2,400 (ViT-L), AdamW with weight decay 0.05, a learning rate warmed up to $3 \times 10^{-4}$ (ViT-B) or $2 \times 10^{-4}$ (ViT-L), and random $224 \times 224$ crops, raised to $384 \times 384$ for fine-tuning. The paper says the learning rate decays “linearly with a rate of 0.85”; the released code multiplies it by 0.9 each epoch (configs/pretrain.yaml; step_lr_schedule in utils.py).

CapFilt

CapFilt (Captioning and Filtering) replaces the web text before the final pretraining (Li et al., 2022, §3.3 and Figure 3):

1. Pretrain a MED on the noisy data: web pairs $(I_w, T_w)$ plus human-annotated pairs $(I_h, T_h)$.
2. Fine-tune two copies on COCO. The captioner is the image-grounded decoder, fine-tuned with LM; the filter is the image-grounded encoder, fine-tuned with ITC and ITM.
3. The captioner writes one synthetic caption $T_s$ per web image.
4. The filter removes every web caption and every synthetic caption whose ITM head predicts “unmatched”.
5. Pretrain a new model on the filtered web pairs, the filtered synthetic pairs and the human-annotated pairs.

The authors interpret this as knowledge distillation through data: the captioner passes on its knowledge as captions, and the filter by removing noisy ones (§2.2). Several ablations on the 14M-image set show what matters:

• Both modules help, and they add up. Fine-tuned COCO retrieval (text R@1 / image R@1) rose from 78.4 / 60.7 without CapFilt to 79.1 / 61.5 with the filter alone, 79.7 / 62.0 with the captioner alone, and 80.6 / 63.1 with both. COCO captioning CIDEr rose from 127.8 to 129.7 and zero-shot NoCaps CIDEr from 102.2 to 105.1 (Table 1).
• Diverse captions beat safe ones. Synthetic captions drawn by nucleus sampling, which samples each token from the smallest set of tokens whose probability exceeds $p = 0.9$ (0.85 and 0.95 gave similar results), outperformed beam search, 80.6 against 79.6 COCO text R@1, although the filter rejected more of them (25% against 19%). The authors suggest that sampled captions carry more new information, while beam search produces common, safe captions (§4.3, Table 2).
• Captioner and filter must be separate. Sharing parameters between them, as in pretraining, lowered COCO text R@1 to 79.8 and the filter rejected only 8% of synthetic captions; the authors attribute this to confirmation bias, the filter accepting the captioner’s own mistakes (§4.4, Table 4).
• Not longer training. Replicating the original web text to the bootstrapped set’s 24.7M texts left results unchanged (78.3 / 60.5), and continuing to train the first model on the bootstrapped data gave less than training a new one (Appendix A, Tables 12–13).
• It scales. On 129M images, CapFilt with ViT-B modules raised COCO text R@1 from 79.6 to 81.9; with ViT-L modules (CapFilt-L) the ViT-L model went from 80.6 to 82.4 (Table 1).

The released bootstrapped datasets are listed in the official repository (README, as of November 2026).

BLIP-2’s Two-Stage Pretraining

Stage 1: representation learning with a frozen image encoder. The Q-Former is attached to the frozen image encoder and trained on image–text pairs with three objectives inspired by BLIP, each with its own self-attention mask between queries and text (Li et al., 2023, §3.2, Figure 2):

• ITC aligns the query outputs $Z = (z_1, \dots, z_{32})$ with the text [CLS] output $t$. A unimodal mask stops queries and text from seeing each other, and the image–text similarity is the best-matching query, $s(I, T) = \max_k z_k^\top t$ on normalized features. Because the frozen encoder leaves room for large batches, BLIP-2 uses in-batch negatives instead of BLIP’s momentum queue. The code adds label smoothing of 0.1 to this loss, which the paper does not mention (blip2_qformer.py).
• Image-grounded text generation (ITG) trains the text transformer to generate the caption. The text cannot see the image encoder directly: under a multimodal causal mask the queries see each other, and each text token sees all queries and the earlier text tokens. The queries are thus forced to extract everything the text needs. A [DEC] token replaces [CLS].
• ITM uses a bidirectional mask, so that queries and text see each other. Each of the 32 query outputs passes through a two-class linear classifier, and the logits are averaged to score the pair. Negatives are mined as in ALBEF and BLIP.

Stage 2: generative learning with a frozen LLM. The Q-Former, still attached to the frozen image encoder, feeds its projected queries to the frozen LLM as soft prompts. With OPT the model is trained with language modeling on the caption; with FlanT5, with prefix language modeling, in which the caption’s first part goes to the LLM’s encoder with the queries and the rest is the decoder’s target (§3.3, Figure 3). The authors argue that stage 1 has already made the queries language-informative, which reduces the burden on the LLM and the catastrophic forgetting that would follow.

Without stage 1, the Q-Former is trained by generation alone, which the authors compare to Flamingo’s Perceiver Resampler. Zero-shot VQAv2 accuracy was then much lower for both LLM types, and with OPT it fell as training went on, which they read as catastrophic forgetting (§4.1, Figure 5).

Data and settings. BLIP-2 reuses BLIP’s 129M images, with a CapFilt variant: the BLIP-large captioner writes 10 captions per web image; these and the web caption are ranked by a CLIP ViT-L/14 image–text similarity; the top two are kept, and one is sampled at each step. Stage 1 runs 250K steps (batch 2,320 for ViT-L, 1,680 for ViT-g) and stage 2 80K steps (batch 1,920 for OPT, 1,520 for FlanT5), with frozen weights in FP16 (BFloat16 for FlanT5), AdamW, a cosine schedule peaking at $10^{-4}$ and $224 \times 224$ images. The largest model, ViT-g with FlanT5-XXL, took under 6 days for stage 1 and under 3 days for stage 2 on one machine with 16 A100 (40 GB) GPUs (§3.4).

Example: CapFilt on noisy synthetic captions

The example builds a toy CapFilt pipeline in PyTorch, trained from scratch with no pretrained weights. The images are $24 \times 24$ pictures of one small or large, red, green, blue or yellow circle, square or triangle in one of four quadrants, captioned by a four-word template such as “small red circle top-left” (96 possible captions). A “human” set has 400 correct pairs. A “web” set has 4,000 images whose captions are 50% wrong: a quarter are replaced by an unrelated caption and a quarter have one attribute changed. The tiny MED has a convolutional patch encoder and two text blocks. Each block holds a bidirectional and a causal self-attention layer, used by the encoder and decoder modes, with cross-attention and feed-forward layers shared as in BLIP. It trains with ITC, ITM on hard negatives and LM with label smoothing 0.1; the momentum encoder and queue are left out. Because the toy has only 96 captions, identical captions in a batch count as positives in ITC and are never drawn as ITM negatives.

A model pretrained on the noisy data is fine-tuned on the human set into a captioner and a filter. The captioner writes one nucleus-sampled caption ($p = 0.9$) per web image, and the filter keeps a caption when its ITM probability of a match exceeds 0.5. The model pretrained on the web text is the baseline, and four new models are pretrained from scratch for the same number of steps: after the filter only, with the captioner only, with both, and on the web images’ true captions. The test is image-to-text R@1 (recall at 1) over all 96 captions on 1,000 clean test images, ranked by ITC similarity without BLIP’s ITM re-ranking. Before the first run, a fixed hand rule with no training recovered colour and quadrant from the pixels for every image and size and shape for 91% of them, so the attributes can be learned but small shapes are not trivial. The script runs in about four and a half minutes on one CPU thread with PyTorch 2.14.1; it takes the seed as its first argument and writes its runtime to standard error.

import sys, time
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

SEED = int(sys.argv[1]) if len(sys.argv) > 1 else 0
STEPS = int(sys.argv[2]) if len(sys.argv) > 2 else 500       # pretraining steps per model
torch.set_num_threads(1)
torch.manual_seed(SEED)
rng = np.random.default_rng(SEED)

# ---- synthetic images and template captions ---------------------------------------
SIZES = ["small", "large"]
COLORS = {"red": (1, .15, .15), "green": (.15, 1, .15), "blue": (.2, .3, 1), "yellow": (1, 1, .15)}
SHAPES = ["circle", "square", "triangle"]
PLACES = ["top-left", "top-right", "bottom-left", "bottom-right"]
SLOTS = [SIZES, list(COLORS), SHAPES, PLACES]                 # caption = one word per slot
WORDS = [w for slot in SLOTS for w in slot]
CLS, ENC, DEC = 0, 1, 2                                       # special tokens
V = 3 + len(WORDS)
ALL_CAPS = [(a, b, c, d) for a in range(2) for b in range(4) for c in range(3) for d in range(4)]
H = 24
yy, xx = np.mgrid[0:H, 0:H] + 0.5

def draw(attr):
    size, color, shape, place = attr
    cy = (6 if place < 2 else 18) + rng.uniform(-1.5, 1.5)
    cx = (6 if place % 2 == 0 else 18) + rng.uniform(-1.5, 1.5)
    r = (3.0 if size == 0 else 5.0) * rng.uniform(0.9, 1.1)
    y, x = yy - cy, xx - cx
    if shape == 0:
        m = x**2 + y**2 < r**2
    elif shape == 1:
        m = (abs(x) < 0.85 * r) & (abs(y) < 0.85 * r)
    else:
        m = (y < 0.8 * r) & (y > -r + 2 * abs(x))
    rgb = np.array(COLORS[SLOTS[1][color]]) * rng.uniform(0.7, 1.0)
    img = 0.1 + m[..., None] * rgb + 0.1 * rng.standard_normal((H, H, 3))
    return img.transpose(2, 0, 1).astype(np.float32)

def tokens(attr):                                             # word ids, offset by slot
    off = np.cumsum([0] + [len(s) for s in SLOTS[:-1]])
    return [3 + int(o) + a for o, a in zip(off, attr)]

def corrupt(attr, kind):
    if kind == "random":                                      # unrelated caption
        new = list(ALL_CAPS[rng.integers(len(ALL_CAPS))])
        while tuple(new) == tuple(attr):
            new = list(ALL_CAPS[rng.integers(len(ALL_CAPS))])
    else:                                                     # one attribute changed
        new, k = list(attr), rng.integers(4)
        new[k] = rng.choice([v for v in range(len(SLOTS[k])) if v != attr[k]])
    return tuple(new)

def make_set(n, noise=0.0):
    attrs = [ALL_CAPS[i] for i in rng.integers(len(ALL_CAPS), size=n)]
    imgs = np.stack([draw(a) for a in attrs])
    kind = rng.choice(["clean", "random", "one-attr"], size=n, p=[1 - noise, noise / 2, noise / 2])
    caps = [a if k == "clean" else corrupt(a, k) for a, k in zip(attrs, kind)]
    return torch.tensor(imgs), torch.tensor([tokens(c) for c in caps]), attrs, kind

# ---- a tiny multimodal mixture of encoder-decoder (MED) -----------------------------
D = 64

class Block(nn.Module):
    """Text block: bi-directional SA (encoder) or causal SA (decoder); CA and FFN shared."""
    def __init__(self):
        super().__init__()
        self.sa_bi = nn.MultiheadAttention(D, 4, batch_first=True)
        self.sa_causal = nn.MultiheadAttention(D, 4, batch_first=True)
        self.ca = nn.MultiheadAttention(D, 4, batch_first=True)
        self.ffn = nn.Sequential(nn.Linear(D, 4 * D), nn.GELU(), nn.Linear(4 * D, D))
        self.n1, self.n2, self.n3 = nn.LayerNorm(D), nn.LayerNorm(D), nn.LayerNorm(D)

    def forward(self, t, img=None, causal=False):
        h = self.n1(t)
        if causal:
            mask = torch.triu(torch.ones(t.shape[1], t.shape[1], dtype=torch.bool), 1)
            t = t + self.sa_causal(h, h, h, attn_mask=mask)[0]
        else:
            t = t + self.sa_bi(h, h, h)[0]
        if img is not None:                                   # image-grounded modes only
            h = self.n2(t)
            t = t + self.ca(h, img, img)[0]
        return t + self.ffn(self.n3(t))

class MED(nn.Module):
    def __init__(self):
        super().__init__()
        self.cnn = nn.Sequential(nn.Conv2d(3, 32, 3, padding=1), nn.GELU(),
                                 nn.Conv2d(32, D, 3, stride=2, padding=1), nn.GELU(),
                                 nn.Conv2d(D, D, 3, stride=2, padding=1))      # 6x6 patch grid
        self.img_pos = nn.Parameter(0.02 * torch.randn(36, D))
        self.tok = nn.Embedding(V, D)
        self.txt_pos = nn.Parameter(0.02 * torch.randn(5, D))
        self.blocks = nn.ModuleList([Block() for _ in range(2)])
        self.img_proj, self.txt_proj = nn.Linear(D, 32), nn.Linear(D, 32)
        self.itm_head, self.lm_head = nn.Linear(D, 2), nn.Linear(D, V)
        self.temp = nn.Parameter(torch.tensor(0.07))

    def image(self, x):
        return self.cnn(x).flatten(2).transpose(1, 2) + self.img_pos         # B x 36 x D

    def text(self, first, words, img=None, causal=False):
        t = self.tok(torch.cat([torch.full((len(words), 1), first), words], 1)) + self.txt_pos
        for b in self.blocks:
            t = b(t, img, causal)
        return t

    def itc_feats(self, img, words):
        u = F.normalize(self.img_proj(img.mean(1)), dim=-1)
        w = F.normalize(self.txt_proj(self.text(CLS, words)[:, 0]), dim=-1)
        return u, w

    def itm_logits(self, img, words):
        return self.itm_head(self.text(ENC, words, img)[:, 0])

    def lm_logits(self, img, words):                         # predict word k from DEC, w_<k
        return self.lm_head(self.text(DEC, words, img, causal=True)[:, :-1])

def losses(m, x, words, use=("itc", "itm", "lm")):
    img = m.image(x)
    out, bs = {}, len(x)
    same = (words[:, None, :] == words[None, :, :]).all(-1).float()        # identical captions
    if "itc" in use or "itm" in use:
        u, w = m.itc_feats(img, words)
        sim = u @ w.T / m.temp.clamp(0.01, 0.5)
        tgt = same / same.sum(1, keepdim=True)                # duplicates count as positives
        out["itc"] = -0.5 * ((F.log_softmax(sim, 1) * tgt).sum(1).mean()
                             + (F.log_softmax(sim.T, 1) * tgt).sum(1).mean())
    if "itm" in use:                                          # ALBEF/BLIP hard negatives
        with torch.no_grad():
            wi2t = F.softmax(sim, 1).masked_fill(same.bool(), 0) + 1e-6 * (1 - same)
            wt2i = F.softmax(sim.T, 1).masked_fill(same.bool(), 0) + 1e-6 * (1 - same)
            neg_t = torch.multinomial(wi2t, 1).squeeze(1)    # a negative caption per image
            neg_i = torch.multinomial(wt2i, 1).squeeze(1)    # a negative image per caption
        logits = torch.cat([m.itm_logits(img, words), m.itm_logits(img, words[neg_t]),
                            m.itm_logits(img[neg_i], words)])
        lab = torch.cat([torch.ones(bs), torch.zeros(2 * bs)]).long()
        out["itm"] = F.cross_entropy(logits, lab)
    if "lm" in use:
        lg = m.lm_logits(img, words)
        out["lm"] = F.cross_entropy(lg.reshape(-1, V), words.reshape(-1), label_smoothing=0.1)
    return sum(out.values())

def train(m, X, T, steps, use=("itc", "itm", "lm"), lr=2e-3, bs=64):
    opt = torch.optim.AdamW(m.parameters(), lr=lr, weight_decay=0.05)
    g = torch.Generator().manual_seed(SEED)
    m.train()
    for _ in range(steps):
        idx = torch.randint(len(X), (bs,), generator=g)
        loss = losses(m, X[idx], T[idx], use)
        opt.zero_grad()
        loss.backward()
        opt.step()
    return m

@torch.no_grad()
def recall_at_1(m, X, attrs):
    """Image-to-text R@1 over all 96 captions by ITC similarity (no ITM re-ranking)."""
    m.eval()
    u, w = m.itc_feats(m.image(X), torch.tensor([tokens(c) for c in ALL_CAPS]))
    pred = (u @ w.T).argmax(1)
    return np.mean([ALL_CAPS[p] == a for p, a in zip(pred.tolist(), attrs)])

@torch.no_grad()
def itm_match(m, X, T):
    m.eval()
    return F.softmax(m.itm_logits(m.image(X), T), 1)[:, 1] > 0.5

@torch.no_grad()
def nucleus_caption(m, X, p=0.9):
    m.eval()
    img, g = m.image(X), torch.Generator().manual_seed(SEED)
    words = torch.zeros(len(X), 0, dtype=torch.long)
    for k in range(4):
        pad = torch.cat([words, torch.zeros(len(X), 4 - k, dtype=torch.long)], 1)
        probs = F.softmax(m.lm_logits(img, pad)[:, k], -1)
        ps, order = probs.sort(1, descending=True)
        keep = ps.cumsum(1) - ps < p                          # smallest set with mass >= p
        ps = ps * keep
        pick = torch.multinomial(ps / ps.sum(1, keepdim=True), 1, generator=g)
        words = torch.cat([words, order.gather(1, pick)], 1)
    return words

# ---- data: a small clean "human" set, a large noisy "web" set, a test set ----------
t0 = time.time()
Xh, Th, _, _ = make_set(400)                                  # like COCO: small and clean
Xw, Tw, Aw, kind = make_set(4000, noise=0.5)                  # like alt-text: half wrong
Xte, _, Ate, _ = make_set(1000)
Tw_true = torch.tensor([tokens(a) for a in Aw])

def pretrain(X, T):
    torch.manual_seed(SEED)
    return train(MED(), X, T, STEPS)

results = {}
base = pretrain(torch.cat([Xw, Xh]), torch.cat([Tw, Th]))
results["web text, no CapFilt"] = recall_at_1(base, Xte, Ate)

# CapFilt: captioner (LM) and filter (ITC + ITM), both fine-tuned from `base` on the human set
cap, filt = MED(), MED()
cap.load_state_dict(base.state_dict()); filt.load_state_dict(base.state_dict())
train(cap, Xh, Th, 300, use=("lm",), lr=5e-4)
train(filt, Xh, Th, 300, use=("itc", "itm"), lr=5e-4)

Ts = nucleus_caption(cap, Xw)
keep_w, keep_s = itm_match(filt, Xw, Tw), itm_match(filt, Xw, Ts)
syn_ok = (Ts == Tw_true).all(1)
print("filter on web captions (kept fraction):")
for k in ["clean", "one-attr", "random"]:
    sel = torch.tensor(kind == k)
    print(f"  {k:9s} {keep_w[sel].float().mean():.2f}   (n = {int(sel.sum())})")
print(f"synthetic captions correct: {syn_ok.float().mean():.2f};  kept: {keep_s.float().mean():.2f};"
      f"  correct among kept: {syn_ok[keep_s].float().mean():.2f}")

sets = {
    "filter only": (torch.cat([Xw[keep_w], Xh]), torch.cat([Tw[keep_w], Th])),
    "captioner only": (torch.cat([Xw, Xw, Xh]), torch.cat([Tw, Ts, Th])),
    "captioner + filter": (torch.cat([Xw[keep_w], Xw[keep_s], Xh]),
                           torch.cat([Tw[keep_w], Ts[keep_s], Th])),
    "true captions (oracle)": (torch.cat([Xw, Xh]), torch.cat([Tw_true, Th])),
}
for name, (X, T) in sets.items():
    results[name] = recall_at_1(pretrain(X, T), Xte, Ate)
print("\nnew model pretrained on        pairs   test image-to-text R@1 (96 captions)")
sizes = {"web text, no CapFilt": len(Xw) + len(Xh), **{k: len(v[0]) for k, v in sets.items()}}
for name, r in results.items():
    print(f"  {name:26s} {sizes[name]:5d}   {r:.3f}")
print(f"\n(seed {SEED}, {STEPS} steps per pretraining, {time.time() - t0:.0f} s)", file=sys.stderr)

filter on web captions (kept fraction):
  clean     0.48   (n = 2003)
  one-attr  0.08   (n = 1004)
  random    0.01   (n = 993)
synthetic captions correct: 0.53;  kept: 0.46;  correct among kept: 0.66

new model pretrained on        pairs   test image-to-text R@1 (96 captions)
  web text, no CapFilt        4400   0.280
  filter only                 1451   0.497
  captioner only              8400   0.369
  captioner + filter          3286   0.426
  true captions (oracle)      4400   0.423

The filter is conservative but accurate. It keeps under half of the correct web captions and almost none of the unrelated ones; one-attribute errors pass slightly more often, which by our reading is because they differ from the truth in a single word. By our arithmetic from the printed fractions, about 91% of the kept web captions are correct, against 50% before filtering. The captioner is weak after fine-tuning on 400 pairs, getting just over half of its captions exactly right, and the filter raises that to two thirds among the captions it keeps. Pretraining on the raw web text gives the lowest R@1. Adding unfiltered synthetic captions helps, and either filtered set helps more, although the filter removes nearly three quarters of the web captions.

Run with seeds 1 and 2 as its first argument, the script agrees that web text < captioner only < filter only: R@1 was 0.124–0.280 on web text, 0.221–0.369 with the captioner only, 0.359–0.497 with the filter only and 0.411–0.426 with both, over seeds 0–2. They disagree on the rest. Captioner plus filter beat filter only in seeds 1 and 2 but not in seed 0, and the true captions gave 0.423–0.461, so the best bootstrapped set came within 0.05 of clean data in seeds 1 and 2 and passed it in seed 0. The filter kept 23–57% of correct web captions, 5–18% of one-attribute errors and 1–8% of unrelated ones.

What the toy cannot show:

• The models are undertrained. At 500 steps even the model trained on true captions retrieves the right caption less than half the time, so absolute values mean little. The filtered sets are smaller, so with a fixed number of steps they are also seen more often, and the filter may keep easier images. The run separates neither effect, and we do not know why seed 0’s filtered set beat clean data.
• Diversity cannot appear. Template captions with four slots have no room for the “diverse and surprising” captions that made nucleus sampling beat beam search in the paper.
• Scale and transfer are missing. BLIP’s captioner and filter start from a model pretrained on 14M images and fine-tuned on COCO, and its gains are measured after fine-tuning on downstream tasks. Here everything is trained from scratch on one synthetic distribution.

Inference

Retrieval. BLIP first ranks all candidates by ITC similarity and then re-scores the top $k$ with the ITM head, $k = 256$ for COCO and $k = 128$ for Flickr30k, following ALBEF (Li et al., 2022, §5.1). In the released code, the final score of a re-ranked candidate is the ITM logit for “matched” plus the ITC similarity (train_retrieval.py). BLIP-2 uses $k = 128$ (Li et al., 2023, §4.4). The image–text retrieval article discusses the cost of this two-stage protocol.

Captioning. BLIP prepends the prompt “a picture of”, which helped slightly, and decodes by beam search with beam size 3 and at most 20 tokens (§5.2, Appendix B). Fine-tuned BLIP-2 uses the prompt “a photo of” with beam size 5 (Table 7).

VQA and other tasks. BLIP treats VQA as answer generation: an encoder reads the image and question, and an answer decoder, fine-tuned with LM, ranks 3,128 candidate answers at inference, at $480 \times 480$ resolution (§5.3, Appendix B). For NLVR2, which asks whether a sentence describes a pair of images, each encoder block gets two cross-attention layers, one per image, initialized from the same weights and merged by averaging (first six layers) or concatenation and projection (last six). For visual dialog, a dialog encoder ranks candidate answers with the ITM loss (§5.4–5.5). For video, BLIP samples 8 frames (retrieval) or 16 (QA) and concatenates their features, ignoring temporal order (§5.6).

Zero-shot generation with BLIP-2. A text prompt follows the visual prompt, so the LLM can follow instructions about the image (§4.1). For zero-shot VQA, the prompts are “Question: {} Answer:” (OPT) and “Question: {} Short answer:” (FlanT5), with beam width 5 and a length penalty of −1, which favours short answers. When BLIP-2 is fine-tuned for VQA, the question is also given to the Q-Former, so the queries can attend to question-relevant regions (§4.3).

Performance

All results are as the authors reported them; “zero-shot” Flickr30k retrieval in both papers means a model fine-tuned on COCO and tested on Flickr30k without further training, which the image–text retrieval article discusses. The CLIP row is the exception: it is CLIP’s own zero-shot result, without COCO fine-tuning.

Retrieval (R@1 in percent; COCO 5K test and Flickr30k 1K test of the Karpathy split):

Model	Pretraining images	COCO fine-tuned TR@1	COCO fine-tuned IR@1	Flickr30k zero-shot TR@1	Flickr30k zero-shot IR@1
CLIP	400M	—	—	88.0	68.7
ALBEF	14M	77.6	60.7	94.1	82.8
BLIP ViT-B	14M	80.6	63.1	94.8	84.9
BLIP ViT-B	129M	81.9	64.3	96.0	85.0
BLIP ViT-B, CapFilt-L	129M	81.2	64.1	96.0	85.5
BLIP ViT-L	129M	82.4	65.1	96.7	86.7
BLIP-2 ViT-L	129M	83.5	66.3	96.9	88.6
BLIP-2 ViT-g	129M	85.4	68.3	97.6	89.7

BLIP rows from Li et al. (2022), Tables 5–6; BLIP-2 rows from Li et al. (2023), Table 5. TR is image-to-text and IR text-to-image retrieval. Fine-tuned on Flickr30k itself, BLIP ViT-L reached TR@1 97.4 and IR@1 87.6 (Li et al., 2022, Table 5). With the same 14M images, BLIP’s average COCO R@1 is 2.7 points above ALBEF’s. BLIP-2’s ablation found that its generation loss also helps retrieval: COCO R@1 84.5 / 67.2 with ITC + ITM and 85.4 / 68.3 with ITG added (Li et al., 2023, Table 6).

Captioning (fine-tuned on COCO with cross-entropy; COCO Karpathy test and the NoCaps validation set, zero-shot):

Model	COCO BLEU-4	COCO CIDEr	NoCaps CIDEr	NoCaps SPICE
BLIP ViT-B, 14M	38.6	129.7	105.1	14.4
BLIP ViT-B, 129M	39.4	131.4	106.3	14.3
BLIP ViT-B, CapFilt-L, 129M	39.7	133.3	109.6	14.7
BLIP ViT-L, 129M	40.4	136.7	113.2	14.8
BLIP-2 ViT-g, OPT-2.7B	43.7	145.8	119.7	15.4
BLIP-2 ViT-g, OPT-6.7B	43.5	145.2	121.0	15.3
BLIP-2 ViT-g, FlanT5-XL	42.4	144.5	121.6	15.8

From Li et al. (2022), Table 7, and Li et al. (2023), Table 3. BLIP-2 fine-tunes the Q-Former and the image encoder but keeps the LLM frozen. The authors compare BLIP ViT-B on 129M images with LEMON-base on 200M images (COCO CIDEr 131.4 against 133.3, NoCaps 106.3 against 106.8) and call it competitive, noting that LEMON needs a pretrained object detector and $800 \times 1333$ inputs, while BLIP uses $384 \times 384$ images without a detector.

Visual question answering. Fine-tuned on VQAv2 (train and val plus Visual Genome questions), BLIP reached 77.54 (test-dev) and 77.62 (test-std) with 14M images and 78.25 and 78.32 with CapFilt-L on 129M images, against 75.84 and 76.04 for ALBEF (Li et al., 2022, Table 8). The text calls the 14M gain over ALBEF “+1.64% on the test set”; the table’s differences are 1.70 (test-dev) and 1.58 (test-std). Fine-tuned BLIP-2 with ViT-g and OPT-6.7B reached 82.19 and 82.30 (Li et al., 2023, Table 4); the ICML version heads these columns “val” and “test-dev”, but its ALBEF and BLIP rows repeat the test-dev and test-std values of the BLIP paper. On NLVR2, BLIP gained little from the web images (82.67 dev and 82.30 test-P with 14M images; 82.48 and 83.08 with 129M), which the authors attribute to the gap between web and downstream data.

Zero-shot VQA, BLIP-2 only (accuracy in percent, VQAv2 test-dev; Li et al., 2023, Table 2):

Model	Trainable parameters	VQAv2 test-dev	OK-VQA test	GQA test-dev
Flamingo-80B	10.2B	56.3	50.6	—
BLIP-2 ViT-L, OPT-2.7B	104M	49.7	30.2	33.9
BLIP-2 ViT-g, OPT-6.7B	108M	52.6	36.4	36.4
BLIP-2 ViT-L, FlanT5-XL	103M	62.3	39.4	44.4
BLIP-2 ViT-g, FlanT5-XXL	108M	65.0	45.9	44.7

The authors observe that a larger image encoder or a larger LLM improved accuracy, and that the instruction-tuned FlanT5 beat OPT. The headline claim, 8.7 points above Flamingo-80B with 54 times fewer trainable parameters, matches the 188M parameters of Table 1, the paper’s overview of zero-shot results; Table 2 lists 103M–108M trainable parameters for the same models, and the paper does not explain the difference. On OK-VQA, BLIP-2 trails Flamingo-80B, which the authors attribute to the greater knowledge of Flamingo’s 70B Chinchilla language model.

Video, zero-shot. On the MSR-VTT 1k test split, BLIP fine-tuned for COCO retrieval reached text-to-video R@1 43.3 without video training, above the 33.9 of ALPRO fine-tuned on MSR-VTT; zero-shot top-1 accuracy was 19.2 on MSRVTT-QA and 35.2 on MSVD-QA (Li et al., 2022, Tables 10–11).

Strengths

• One model for understanding and generation. BLIP’s shared encoder–decoder transfers to retrieval, captioning, VQA, NLVR2 and dialog, at the cost of one image encoder pass and three cheap text passes per pair.
• Data cleaning as a method. CapFilt improved every downstream measure in its ablation and worked at 129M images and with a larger backbone; the bootstrapped data was released.
• Low training cost for BLIP-2. Only the Q-Former and a projection are trained, and the method can reuse newer frozen encoders and LLMs: a stronger image encoder or LLM gave better zero-shot VQA in every comparison of Table 2 but one (with FlanT5-XL, GQA fell from 44.4 with ViT-L to 44.2 with ViT-g).

Limitations

• Single image–text pairs. BLIP-2 did not improve on VQA when given in-context examples, which the authors attribute to pretraining on one image–text pair per sample, unlike Flamingo’s interleaved web pages (Li et al., 2023, §5).
• Inherited LLM failures. BLIP-2 can output wrong knowledge, follow a wrong line of reasoning or give outdated information, and it inherits the LLM’s risks, such as offensive language, social bias and leaking private information (§5).
• Filter and captioner are trained on COCO. By our reading, CapFilt pushes the corpus towards captions that a COCO-trained filter accepts and a COCO-trained captioner writes, so the bootstrapped data inherits COCO’s style and its blind spots, as CLIP-filtered data inherits CLIP’s.
• No temporal model for video. BLIP’s video transfer concatenates frame features and ignores order (Li et al., 2022, §5.6).
• Short answers. Zero-shot VQA prompts and length penalties favour short answers that match the benchmark’s annotation style; the visual question answering article discusses how this affects comparisons.

Historical Importance

BLIP (2022) unified the contrastive, matching and generative objectives of vision–language pretraining in one shared text transformer, and showed that the quality of web captions, not only their quantity, limits pretraining. BLIP-2 (2023) showed that a small trainable module between a frozen image encoder and a frozen LLM could beat Flamingo-80B on zero-shot VQAv2 and report the highest zero-shot NoCaps CIDEr among the models it compared, with a fraction of the trainable parameters. Its Q-Former is one of the connector designs of vision–language models, next to Flamingo’s resampler and the projector of LLaVA, whose authors chose a projection because it is lightweight, leaving the Q-Former as an alternative, and InstructBLIP carried it into instruction tuning.

Variants

• BLIP backbones and data (2022). ViT-B and ViT-L models, pretrained on 14M or 129M images, with CapFilt modules of either size (CapFilt-L uses ViT-L modules to bootstrap data for a ViT-B model) (Li et al., 2022).
• BLIP-2 (2023). CLIP ViT-L/14 or EVA-CLIP ViT-g/14 with OPT-2.7B, OPT-6.7B, FlanT5-XL or FlanT5-XXL, and a retrieval model without an LLM (Li et al., 2023).
• InstructBLIP (2023). Dai et al. (2023) instruction-tuned BLIP-2. They converted 26 public datasets in 11 task categories into instruction format with 10–15 templates per task, tuned on 13 (held-in) and tested zero-shot on the other 13, including four entirely held-out task categories. Their instruction-aware Q-Former also receives the instruction tokens, which interact with the queries through self-attention, so that the extracted visual features depend on the task. Only the Q-Former is fine-tuned; the ViT-g/14 encoder and the LLM (FlanT5-XL, FlanT5-XXL, Vicuna-7B or Vicuna-13B, the last two after BLIP-2 pretraining with Vicuna by the authors) stay frozen, with 32 queries. Datasets are sampled in proportion to the square root of their size, with manual adjustments. InstructBLIP FlanT5-XL improved on BLIP-2 FlanT5-XL by 15.0% on average (relative) over the held-out datasets, and removing the instruction input to the Q-Former lowered every held-out score in the ablation, for example ScienceQA (image context) from 70.4 to 63.4 (Tables 1–2). VQAv2 is one of InstructBLIP’s training datasets, so its VQAv2 results are not zero-shot.

Implementations

The official BLIP code (salesforce/BLIP) provides pretraining and fine-tuning code, checkpoints and the bootstrapped datasets, and was tested on PyTorch 1.10; as of November 2026 the repository is archived and its README advises against using it for production or sensitive purposes. BLIP, BLIP-2 and InstructBLIP are implemented in Salesforce’s LAVIS library, which provides the BLIP-2 checkpoints (for example blip2_opt with pretrain_opt2.7b, blip2_t5 with pretrain_flant5xl, and blip2 for feature extraction and retrieval) and InstructBLIP (blip2_vicuna_instruct, blip2_t5_instruct); as of November 2026 it is also archived. Both repositories are released under the BSD-3-Clause licence.