Flamingo

Flamingo is a family of vision–language models from DeepMind that read text interleaved with images or videos and write text. Jean-Baptiste Alayrac and 26 co-authors presented it at NeurIPS 2022 (the model card in the paper’s appendix dates the model to March 2022). It keeps two pretrained networks frozen, an NFNet image encoder trained contrastively on image–text pairs and a Chinchilla language model, and trains only what connects them: a Perceiver Resampler that turns each image or video into 64 visual tokens, and new cross-attention layers inserted between the frozen language-model layers, each gated by $\tanh(\alpha)$ with $\alpha$ initialized at zero. Trained on 43 million web pages with their images in place, plus image–text and video–text pairs, a single Flamingo model learns a new task from a handful of examples placed in its prompt, the way GPT-3 does for text. Of the 16 benchmarks the paper reports, the largest model, Flamingo-80B, set a new few-shot state of the art on the 9 with published few-shot results and, with 32 examples, beat the fine-tuned state of the art on 6 (Alayrac et al., 2022, Figure 2).

Problem Addressed

In 2022 the usual way to adapt a vision model to a task was to pretrain it and then fine-tune it on thousands of labelled examples, with per-task hyperparameter tuning. Contrastive models such as CLIP removed the fine-tuning for classification, but only score the similarity of an image and a text, so they cannot caption an image or answer an open question. Models that generate text conditioned on images existed, but, in the authors’ assessment, had not done well with little data (Alayrac et al., 2022, §1).

Large language models had shown another route: few-shot in-context learning, where a few input–output examples in the prompt define the task and the model completes the next one. Flamingo carries that interface over to vision. Doing so required three things for which the authors propose architectural innovations: bridging a pretrained vision-only model and a pretrained language-only model without destroying what either had learned, handling arbitrarily interleaved sequences of images and text, and accepting both images and videos (abstract and §1). It also required training data in which images and text interleave as they do in a few-shot prompt, which ordinary image–caption datasets are not.

One earlier system that the paper compares against on few-shot visual question answering is Frozen, by Tsimpoukelli and colleagues (2021, as the Flamingo paper cites it), which the paper groups with other work that kept a pretrained language model frozen; it is the previous best few-shot result on VQAv2 in the paper’s Table 1, 38.2% with 4 shots.

Architecture

Flamingo models the probability of text $y$ given the images and videos $x$ interleaved with it as a product of next-token probabilities,

$$p(y \mid x) = \prod_{\ell=1}^{L} p\left(y_\ell \mid y_{<\ell},\, x_{\le \ell}\right),$$

where $x_{\le \ell}$ is the set of images or videos that precede token $y_\ell$ in the sequence (Eq. 1). The text passes through a frozen decoder-only transformer language model; each image or video passes through the frozen vision encoder and the Perceiver Resampler, which return a fixed set of visual tokens; and new GATED XATTN-DENSE blocks, inserted before frozen language-model blocks, let the text tokens attend to those visual tokens (Figure 3). The paper writes the block name in capitals; this article keeps that spelling.

Vision Encoder

The vision encoder is a Normalizer-Free ResNet, NFNet-F6 (§2.1). NFNets are ResNets designed to train without batch normalization, helped by adaptive gradient clipping (Brock et al., 2021, abstract). DeepMind pretrained it from scratch with a CLIP-style two-term contrastive loss, image-to-text and text-to-image with a learnable temperature, against a BERT text encoder on ALIGN and LTIP image–text pairs, at $288 \times 288$ resolution, a 1,376-dimensional joint space and batches of 16,384, for 1.2 million updates on 512 TPUv4 chips; only the vision weights are kept (Appendix B.1.3). Flamingo uses the output of the final stage, a 2D grid of features flattened into a sequence. Video frames are sampled at 1 frame per second and encoded independently, and learned temporal embeddings are added to each frame’s features (§2.1).

The encoder stays frozen during Flamingo training. In the ablations, training it from scratch cost 9.3 points of the overall score, and fine-tuning it from the contrastive weights cost 2.6 points while raising the compute (Appendix B.3.1, Table 10). Replacing NFNet-F6 with the public CLIP ViT-L/14 at 224 resolution cost 5.8 points, and a smaller NFNet-F0 8.0 points (§3.3, Table 3).

Perceiver Resampler

The language model should see the same small number of visual tokens per image, whatever the resolution or number of frames. The Perceiver Resampler does this with a predefined set of learned latent queries that cross-attend to the flattened visual features $X_f$, as in the Perceiver of Jaegle et al. (2021) and DETR’s object queries (§2.1); the Cross-Attention article explains how the Perceiver’s latent bottleneck makes attention cost grow linearly with the number of inputs.

With latents $X \in \mathbb{R}^{R \times d}$ and features from $T$ frames of $S$ spatial positions each, each of the resampler’s layers computes, following the pseudo-code of the paper’s Figure 5,

$$X \leftarrow X + \operatorname{Attn}\!\left(q = X,\; kv = [X_f;\, X]\right), \qquad X \leftarrow X + \operatorname{FFW}(X),$$

with $X_f \in \mathbb{R}^{TS \times d}$ the features plus temporal embeddings, flattened over time and space. Two details differ from Perceiver and DETR: the keys and values include the latents themselves, concatenated with the visual features, which the authors found slightly better, and there are no spatial position encodings, only temporal ones. The authors saw no gain from spatial encodings and suggest that convolutional features already encode position in their channels (Appendix A.1.1). The output has as many tokens as there are latents: 64 in every Flamingo model (§2.1). The resampler has 6 layers of width 1,536 with 16 heads of dimension 96 and about 200M parameters (194M in Table 5), the same for all three model sizes (Appendix A.1.4, B.1.1).

Video uses the same module. Training clips have 8 frames sampled at 1 frame per second; at inference Flamingo reads 30 frames at 3 frames per second, by linearly interpolating the learned temporal embeddings (Appendix B.1.2). An image is treated as a one-frame video.

In the ablations, the resampler beat an MLP and a plain transformer of similar parameter budget, which were also slower (overall score 70.7 against 66.6 and 66.7; §3.3, Table 3). Of three resampler sizes, the medium one was best (Small 67.9, Large 69.0); scaling the resampler together with the language model made training unstable, so the authors kept the medium size for all models (Appendix B.3.1, Table 10).

GATED XATTN-DENSE Layers

The frozen language model is conditioned on vision by new blocks inserted between its layers and trained from scratch (§2.2, Figure 4). Each block has a cross-attention layer, whose queries come from the text and whose keys and values come from the visual tokens, followed by a dense feed-forward layer; both outputs are multiplied by $\tanh$ of a learnable scalar before the residual addition:

$$y \leftarrow y + \tanh(\alpha_{\text{xattn}})\, \operatorname{Attn}\!\left(q = y,\; kv = x\right), \qquad y \leftarrow y + \tanh(\alpha_{\text{dense}})\, \operatorname{FFW}(y),$$

after which the frozen self-attention and feed-forward layers of the language-model block run as before. Because each $\alpha$ is initialized at 0, every new block adds exactly zero at initialization and the model reproduces the pretrained language model, which the authors credit with better stability and final performance (§2.2). The paper cites LSTM gating for the tanh gate and ReZero for the zero initialization. Without the gating, the overall score fell by 4.2 points (70.7 to 66.5), and the authors observed training instabilities (§3.3, Table 3).

A consequence the paper does not spell out, in our reading: at initialization the gradient reaches only the $\alpha$ scalars, since the new layers’ outputs are multiplied by $\tanh(0) = 0$; the cross-attention and feed-forward weights, and the resampler behind them, start receiving gradient once the gates move away from zero. The example below checks this. The paper plots the gate values over training for Flamingo-3B: their absolute values grow quickly from zero at all 24 layers and appear to grow with depth, though the authors caution that activation scales before the gates may also vary with depth (Appendix A.1.2, Figure 6).

The trainable blocks use squared-ReLU activations, which the authors found better than the GeLU of the frozen language model (Appendix A.1.4). Against alternatives, GATED XATTN-DENSE beat the vanilla cross-attention of the original Transformer decoder (66.9) and GRAFTING, which leaves the language model untouched and learns a separate stack of self- and cross-attention layers on top of its output (63.1) (§3.3, Table 3).

How often to insert the blocks trades expressivity against compute. Inserting one before every language-model layer was best in the Flamingo-3B ablation; every fourth layer reduced the overall score by 1.9 points but sped training up by 66%, and a single block in the middle cost 10.9 points (§3.3, Table 3). The final models use every layer (3B), every fourth (9B) and every seventh (80B), as many as the hardware allowed (§3.3, Appendix B.1.1).

Per-Image Cross-Attention Masking

Interleaved inputs hold several images. Rather than letting each text token attend to all preceding images, Flamingo masks the text-to-image cross-attention so that each text token sees only the visual tokens of the image or video that appeared last before it (§2.3). Appendix A.1.3 formalizes this with a function $\varphi: [1, L] \to [0, N]$ that gives, for each text position, the index of the last preceding image (0 if none). Earlier images still influence a token through the frozen language model’s self-attention over the preceding text (§2.3). The image tag token itself belongs to its image: in the paper’s Figure 7, $\varphi$ switches from 0 to 1 at the first <image> tag.

This scheme generalizes across the number of images. Training sequences from web pages carry at most 5 images, yet the models use up to 32 image–text pairs at evaluation (§2.3, §3.1). Letting tokens attend to all previous images instead scored 7.2 points lower (63.5 against 70.7). Explicit image indices in the tags, or learned per-image index embeddings, were less robust when the number of images changed between training and test (Appendix B.3.1, Table 10).

Model Sizes

Model	Frozen LM	Frozen vision	GATED XATTN-DENSE (trainable)	Resampler (trainable)	Total
Flamingo-3B	1.4B (24 layers, width 2,048)	435M	1.2B, before every layer	194M	3.2B
Flamingo-9B	7.1B (40 layers, width 4,096)	435M	1.6B, every 4th layer	194M	9.3B
Flamingo-80B	70B Chinchilla (80 layers, width 8,192)	435M	10B, every 7th layer	194M	80B

Parameter counts from Table 5 and layer sizes from Table 4 of the appendix. The text of Appendix B.1.1 gives 1.4B and 1.8B additional learned parameters for the 3B and 9B models, which in our reading is the cross-attention blocks plus the resampler. The three language models come from the Chinchilla work of Hoffmann et al. (2022), whose 70B model was trained on 1.4 trillion tokens of MassiveText with 80 layers, 64 heads and width 8,192 (its §4, Table 3). The paper calls the largest model simply “Flamingo”; this article writes Flamingo-80B. About 90% of its weights are frozen (Appendix B.3.1).

Training

Data

Flamingo trains on a mixture of three kinds of web data, none annotated for machine learning (§2.4, Appendix A.3).

• M3W (MultiModal MassiveWeb): text and images extracted from the HTML of about 43 million web pages, with each image’s position taken from the page’s Document Object Model. An <image> tag marks each image in the plain text, and a learned end-of-chunk token <EOC> precedes each image and ends the document. Each training example is a random 256-token subsequence with up to its first 5 images (§2.4). Pages were filtered for English, explicit content and text quality, and images smaller than 64 pixels on a side, with aspect ratios beyond 3, or of obviously low quality, were removed (Appendix A.3.1). M3W holds 185 million individual images (Appendix A.3.4).
• Image–text pairs: ALIGN, 1.8 billion images with alt-text, and LTIP (Long Text & Image Pairs), 312 million images with longer descriptions collected from fewer than ten websites; ALIGN texts average 12.4 tokens, LTIP texts 20.5 (Appendix A.3.3).
• Video–text pairs: VTP (Video & Text Pairs), 27 million short videos of about 22 seconds on average with sentence descriptions (§2.4).

Paired examples are formatted like M3W, with <image> before the caption and <EOC> after it. Which image a text token should attend to on a web page is ambiguous: text may describe the image that follows it or the one before it. During training, each sampled page attends to the next image with probability $p_{\text{next}} = 1/2$ and to the previous one otherwise; this gave a small advantage over always choosing one (Appendix A.3.2, B.3.1).

Objective

The loss is a weighted sum, over the $M$ datasets, of the expected negative log-likelihood of the text given its visual inputs (Eq. 2):

$$\mathcal{L} = \sum_{m=1}^{M} \lambda_m \, \mathbb{E}_{(x, y) \sim \mathcal{D}_m}\left[-\sum_{\ell=1}^{L} \log p\left(y_\ell \mid y_{<\ell},\, x_{\le \ell}\right)\right].$$

The weights are 1.0 for M3W, 0.2 for ALIGN, 0.2 for LTIP and 0.03 for VTP, found at small scale and kept fixed (Appendix B.1.2); the authors call tuning them key to performance (§2.4). Gradients are accumulated over all datasets for each update, which beat a round-robin schedule (overall score 70.7 against 62.9; Table 3).

Procedure

All models use AdamW with global norm clipping at 1, no weight decay for the resampler and 0.1 for the other trainable parameters, and a learning rate warmed up linearly to $10^{-4}$ over 5,000 steps and then held constant; decaying it did not help. Training runs for 500k steps. Images are resized to $320 \times 320$, larger than the encoder’s contrastive pretraining resolution, which costs little because no gradient flows through the frozen encoder; random flips and colour augmentation are applied, and paired-dataset captions get a leading space with probability 0.5 to make the model invariant to a tokenizer artifact (Appendix B.1.2). The implementation used JAX and Haiku on TPUv4; Flamingo-80B trained on 1,536 chips for 15 days with 16-way model parallelism (Appendix B.1.2).

What the ablations show

The paper’s ablations train Flamingo-3B with a shorter schedule and smaller batches and evaluate with 4 shots on the validation subsets of five development benchmarks; the overall score divides each benchmark’s score by the state of the art in Table 1 and averages (§3.3). Relative to the baseline’s 70.7:

• Data mixture. Removing M3W dropped the score to 53.4, removing image–text pairs to 60.9, and removing video–text pairs to 67.3, with the losses on video tasks. Replacing the image–text pairs with LAION-400M gave 66.4 (Table 3).
• Freezing. Training the language model from scratch gave 57.8; fine-tuning the pretrained one gave 62.7, which the authors read as catastrophic forgetting (§3.3). Co-training on MassiveText, the language model’s pretraining data, did not beat freezing (68.6 from the pretrained model, 55.9 from scratch; Appendix B.3.1).
• Language-model pretraining. A frozen model pretrained on C4 instead of MassiveText scored 62.8, with the losses concentrated on question answering (Appendix B.3.1). The text names OKVQA, VQAv2 and MSVDQA, but Table 10 prints 60.6 for MSVDQA in that row, above the baseline’s 36.3, a value that by our arithmetic also does not fit the row’s overall score.

Example: a zero-initialized gate and a fixed-size resampler

The script builds a toy Flamingo from scratch in PyTorch 2.14.1 and checks the mechanics above on synthetic images; no pretrained weights are used. The images and the frozen random encoder are those of the Vision–Language Model article’s tiny VLM, which measures question answering; this script measures the connector’s mechanics. Each $32 \times 32$ image shows one red, green or blue circle, square or triangle in the left or right half, and each sequence interleaves two image–caption chunks, <BOS> <image> a red square on the left <EOC> <image> a .... The frozen vision encoder is two random, never-trained convolutions giving a $4 \times 4$ grid of 64-dimensional features. A two-block causal transformer is first trained on the same sequences as text only, so it learns the caption grammar but must guess the attributes, and is then frozen. The toy Flamingo adds a Perceiver-style resampler (8 latents, keys and values $[X_f; X]$) and a gated cross-attention block before each of the two language-model blocks, with the per-image mask; only the new parts are trained, with Adam and next-token cross-entropy for 800 steps. Three variants are trained: gated without spatial position embeddings (Flamingo’s choice), gated with them (the paper leaves the choice open for encoders other than its CNN), and ungated, with the gate fixed at 1. A last part feeds synthetic videos of 1 to 30 frames to an untrained resampler with temporal embeddings for 8 frames, interpolated as in Flamingo. The script took 77 to 96 seconds on one CPU thread in our runs and gave identical output on four runs.

import sys
import torch
import torch.nn as nn
import torch.nn.functional as F

STEPS = int(sys.argv[1]) if len(sys.argv) > 1 else 800
torch.manual_seed(0)
torch.set_num_threads(1)
S, D, R = 32, 64, 8                       # image size, model width, resampler latents
COLORS = {"red": (1., .15, .15), "green": (.15, 1., .15), "blue": (.15, .15, 1.)}
SHAPES, SIDES = ["circle", "square", "triangle"], ["left", "right"]
VOCAB = ["<BOS>", "<image>", "<EOC>", "a", "on", "the"] + list(COLORS) + SHAPES + SIDES
IDX = {w: i for i, w in enumerate(VOCAB)}
yy, xx = torch.meshgrid(torch.arange(S) + .5, torch.arange(S) + .5, indexing="ij")

def draw(c, s, d):
    """A 32x32 RGB image with one coloured shape in the left or right half."""
    cx = (8. if d == "left" else 24.) + torch.empty(1).uniform_(-3, 3)
    cy = 16. + torch.empty(1).uniform_(-6, 6)
    r = torch.empty(1).uniform_(4, 6)
    x, y = xx - cx, yy - cy
    if s == "circle":   m = x ** 2 + y ** 2 < r ** 2
    elif s == "square": m = (x.abs() < .85 * r) & (y.abs() < .85 * r)
    else:               m = (y < .8 * r) & (y > -r + 2 * x.abs())
    return .1 + m[None] * torch.tensor(COLORS[c])[:, None, None] + .05 * torch.randn(3, S, S)

# Frozen "vision encoder": two random, never-trained convolutions -> 4x4 grid of 64-d features.
encoder = nn.Sequential(nn.Conv2d(3, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
                        nn.Conv2d(32, 64, 3, padding=1), nn.ReLU(), nn.MaxPool2d(4)).requires_grad_(False)

def make_data(n):
    """n web-page-like sequences, each interleaving two image-caption chunks."""
    toks, imgs = [], []
    for _ in range(n):
        seq = ["<BOS>"]
        for _ in range(2):
            c, s, d = (list(COLORS)[torch.randint(3, (1,))], SHAPES[torch.randint(3, (1,))],
                       SIDES[torch.randint(2, (1,))])
            imgs.append(draw(c, s, d))
            seq += ["<image>", "a", c, s, "on", "the", d, "<EOC>"]
        toks.append([IDX[w] for w in seq])
    with torch.no_grad():
        feats = encoder(torch.stack(imgs)).flatten(2).transpose(1, 2)    # (2n, 16, 64)
    return torch.tensor(toks), feats.reshape(n, 2, 16, 64)

def attention(q, k, v, mask=None, heads=4):
    B, Lq, _ = q.shape
    q, k, v = (t.reshape(B, t.shape[1], heads, -1).transpose(1, 2) for t in (q, k, v))
    s = q @ k.transpose(-1, -2) / q.shape[-1] ** .5
    if mask is not None:                                   # mask: (B, Lq, Lk), True = visible
        s = s.masked_fill(~mask[:, None], -1e9)
    a = s.softmax(-1)
    if mask is not None:                                   # a token with nothing visible reads nothing
        a = a * mask.any(-1)[:, None, :, None]
    return (a @ v).transpose(1, 2).reshape(B, Lq, -1)

class Attn(nn.Module):
    def __init__(self):
        super().__init__()
        self.q, self.k, self.v, self.o = (nn.Linear(D, D, bias=False) for _ in range(4))
    def forward(self, x, ctx, mask=None):
        return self.o(attention(self.q(x), self.k(ctx), self.v(ctx), mask))

def ffw():
    return nn.Sequential(nn.Linear(D, 4 * D), nn.GELU(), nn.Linear(4 * D, D))

class LMBlock(nn.Module):                                  # pre-norm causal transformer block
    def __init__(self):
        super().__init__()
        self.n1, self.att, self.n2, self.ff = nn.LayerNorm(D), Attn(), nn.LayerNorm(D), ffw()
    def forward(self, h):
        L = h.shape[1]
        causal = torch.ones(L, L, dtype=torch.bool).tril().expand(h.shape[0], L, L)
        h = h + self.att(self.n1(h), self.n1(h), causal)
        return h + self.ff(self.n2(h))

class TinyLM(nn.Module):
    def __init__(self):
        super().__init__()
        self.emb, self.pos = nn.Embedding(len(VOCAB), D), nn.Parameter(torch.randn(17, D) * .02)
        self.blocks = nn.ModuleList([LMBlock(), LMBlock()])
        self.norm, self.head = nn.LayerNorm(D), nn.Linear(D, len(VOCAB))

class GatedXAttnDense(nn.Module):
    def __init__(self, gated=True):
        super().__init__()
        self.gated = gated
        self.n1, self.xatt, self.n2, self.ff = nn.LayerNorm(D), Attn(), nn.LayerNorm(D), ffw()
        self.alpha_xattn = nn.Parameter(torch.zeros(1))    # tanh(0) = 0 at initialization
        self.alpha_dense = nn.Parameter(torch.zeros(1))
    def gates(self):
        if not self.gated:
            return 1., 1.
        return self.alpha_xattn.tanh(), self.alpha_dense.tanh()
    def forward(self, y, vis, mask, keep=None):
        g1, g2 = self.gates()
        xa = self.xatt(self.n1(y), vis, mask)
        if keep is not None:
            keep.append(xa)
        y = y + g1 * xa
        return y + g2 * self.ff(self.n2(y))

class Resampler(nn.Module):
    """Perceiver-style: R learned latents attend to [features, latents]."""
    def __init__(self, spatial_pos=False, max_frames=None):
        super().__init__()
        self.inp = nn.Linear(64, D)
        self.latents = nn.Parameter(torch.randn(R, D) * .02)
        self.space = nn.Parameter(torch.randn(16, D) * .02) if spatial_pos else None
        self.time = nn.Parameter(torch.randn(max_frames, D) * .02) if max_frames else None
        self.layers = nn.ModuleList([nn.ModuleList([nn.LayerNorm(D), Attn(), nn.LayerNorm(D), ffw()])
                                     for _ in range(2)])
        self.norm = nn.LayerNorm(D)
    def forward(self, f):                                  # f: (B, T, 16, 64) frames of features
        x = self.inp(f)
        if self.space is not None:
            x = x + self.space
        if self.time is not None:                          # learned temporal embeddings,
            t = self.time.T[None]                          # interpolated if T differs
            if f.shape[1] != t.shape[-1]:
                t = F.interpolate(t, size=f.shape[1], mode="linear", align_corners=True)
            x = x + t[0].T[:, None, :]
        x = x.flatten(1, 2)                                # (B, T*16, D)
        lat = self.latents.expand(f.shape[0], R, D)
        for n1, att, n2, ff in self.layers:
            lat = lat + att(n1(lat), n1(torch.cat([x, lat], 1)))
            lat = lat + ff(n2(lat))
        return self.norm(lat)                              # (B, R, D) whatever T is

def image_mask(tokens, n_img=2):
    """Each text token sees only the R tokens of the latest image at or before it."""
    phi = (tokens == IDX["<image>"]).cumsum(1)             # 0 = no image yet
    img_of_key = torch.arange(n_img).repeat_interleave(R) + 1
    return phi[:, :, None] == img_of_key[None, None, :]    # (B, L, n_img*R)

def lm_forward(lm, tokens, xblocks=None, vis=None, mask=None, keep=None):
    h = lm.emb(tokens) + lm.pos[: tokens.shape[1]]
    for i, blk in enumerate(lm.blocks):
        if xblocks is not None:
            h = xblocks[i](h, vis, mask, keep)
        h = blk(h)
    return lm.head(lm.norm(h))

class Flamingo(nn.Module):
    def __init__(self, lm, gated=True, spatial_pos=False):
        super().__init__()
        self.lm = lm                                       # frozen, shared
        self.resampler = Resampler(spatial_pos)
        self.xblocks = nn.ModuleList([GatedXAttnDense(gated) for _ in lm.blocks])
    def forward(self, tokens, feats, keep=None, mask=None):
        B = tokens.shape[0]
        vis = self.resampler(feats.flatten(0, 1)[:, None]).reshape(B, -1, D)   # images resampled one by one
        mask = image_mask(tokens) if mask is None else mask
        return lm_forward(self.lm, tokens, self.xblocks, vis, mask, keep)

def next_token_loss(logits, tokens):
    return F.cross_entropy(logits[:, :-1].reshape(-1, len(VOCAB)), tokens[:, 1:].reshape(-1))

# 1. "Pretrain" the language model on text only, then freeze it.
tr_tok, tr_feat = make_data(4000)
te_tok, te_feat = make_data(1000)
lm = TinyLM()
opt = torch.optim.Adam(lm.parameters(), lr=3e-3)
for step in range(400):
    i = torch.randint(len(tr_tok), (64,))
    loss = next_token_loss(lm_forward(lm, tr_tok[i]), tr_tok[i])
    opt.zero_grad(); loss.backward(); opt.step()
lm.requires_grad_(False)
frozen = {k: v.clone() for k, v in lm.state_dict().items()}

# Attribute positions: logits at position p predict token p+1.
ATTR = {"colour": [2, 10], "shape": [3, 11], "side": [6, 14]}   # positions of the preceding token
def accuracy(logits, tokens):
    pred = logits.argmax(-1)
    return {k: (pred[:, p] == tokens[:, [q + 1 for q in p]]).float().mean().item() for k, p in ATTR.items()}

with torch.no_grad():
    acc = accuracy(lm_forward(lm, te_tok), te_tok)
print("frozen LM, text only   " + "  ".join(f"{k} {v:.2f}" for k, v in acc.items()))

# 2. Add the resampler and gated cross-attention; train only the new parts.
variants = [("gated, no spatial pos", True, False), ("gated, spatial pos", True, True),
            ("ungated, no spatial pos", False, False)]
models = {}
for name, gated, spatial in variants:
    torch.manual_seed(1)
    model = Flamingo(lm, gated, spatial)
    with torch.no_grad():
        dev = (model(te_tok[:200], te_feat[:200]) - lm_forward(lm, te_tok[:200])).abs().max().item()
    print(f"\n[{name}] max |logit change| vs frozen LM at init: {dev:.3g}")
    new = [p for n, p in model.named_parameters() if not n.startswith("lm.")]
    opt = torch.optim.Adam(new, lr=1e-3)
    for step in range(STEPS + 1):
        i = torch.randint(len(tr_tok), (64,))
        loss = next_token_loss(model(tr_tok[i], tr_feat[i]), tr_tok[i])
        opt.zero_grad(); loss.backward()
        if step == 0 and gated:
            ga = [abs(b.alpha_xattn.grad.item()) for b in model.xblocks]
            other = max(p.grad.abs().max().item() for n, p in model.named_parameters()
                        if p.grad is not None and "alpha" not in n and not n.startswith("lm."))
            print(f"  step 0 grads: |d/d alpha_xattn| per layer {ga[0]:.3g}, {ga[1]:.3g};"
                  f" largest grad of any other new weight {other:.3g}")
        opt.step()
        if gated and step % 300 == 0:
            g = [f"{b.alpha_xattn.tanh().item():+.2f}/{b.alpha_dense.tanh().item():+.2f}" for b in model.xblocks]
            print(f"  step {step:4d} loss {loss.item():.3f}  tanh gates (xattn/dense) layer1 {g[0]}  layer2 {g[1]}")
    with torch.no_grad():
        acc = accuracy(model(te_tok, te_feat), te_tok)
    print("  held-out accuracy     " + "  ".join(f"{k} {v:.2f}" for k, v in acc.items()))
    models[name] = model

same = all(torch.equal(frozen[k], v) for k, v in lm.state_dict().items())
print(f"\nfrozen LM weights unchanged after all training: {same}")

# 3. Per-image masking: alter only image 1 and look at the tokens of chunk 2.
model = models["gated, no spatial pos"]
f2 = te_feat[:200].clone()
f2[:, 0] = te_feat[200:400, 0]                             # swap in different first images
with torch.no_grad():
    k1, k2 = [], []
    l1, l2 = model(te_tok[:200], te_feat[:200], k1), model(te_tok[:200], f2, k2)
    chunk2 = slice(9, 17)
    xa = [(a[:, chunk2] - b[:, chunk2]).abs().max().item() for a, b in zip(k1, k2)]
    lo = (l1[:, chunk2] - l2[:, chunk2]).abs().max().item()
print(f"image 1 changed -> on chunk 2, cross-attention output changes by {xa[0]:.3g} (layer 1),"
      f" {xa[1]:.3g} (layer 2); logits by {lo:.3g}")

# 4. Text with no image at all, after training.
no_img = torch.zeros(200, 17, 2 * R, dtype=torch.bool)
with torch.no_grad():
    d = (model(te_tok[:200], te_feat[:200], mask=no_img) - lm_forward(lm, te_tok[:200])).abs().max().item()
print(f"no visible image, trained model vs frozen LM: max |logit change| {d:.3g}")

# 5. Resampler on video: a fixed token count for any number of frames.
res = Resampler(max_frames=8)
print("\nframes  visual features  resampler output  scores per text token (resampled / all features)")
for T in [1, 4, 8, 30]:
    frames = []
    for t in range(T):                                     # a red square moving left to right
        img = torch.full((3, S, S), .1)
        x0 = int(4 + 20 * t / max(T - 1, 1))
        img[0, 12:20, x0:x0 + 8] = 1.
        frames.append(img)
    with torch.no_grad():
        f = encoder(torch.stack(frames)).flatten(2).transpose(1, 2)[None]   # (1, T, 16, 64)
        out = res(f)
    print(f"{T:6d}  {T * 16:15d}  {str(tuple(out.shape[1:])):>16s}  {R:>12d} / {T * 16}")

frozen LM, text only   colour 0.34  shape 0.33  side 0.48

[gated, no spatial pos] max |logit change| vs frozen LM at init: 0
  step 0 grads: |d/d alpha_xattn| per layer 0.000513, 0.000105; largest grad of any other new weight 0
  step    0 loss 0.365  tanh gates (xattn/dense) layer1 -0.00/-0.00  layer2 -0.00/-0.00
  step  300 loss 0.217  tanh gates (xattn/dense) layer1 -0.07/-0.09  layer2 -0.08/-0.12
  step  600 loss 0.071  tanh gates (xattn/dense) layer1 -0.08/-0.11  layer2 -0.10/-0.21
  held-out accuracy     colour 1.00  shape 0.83  side 1.00

[gated, spatial pos] max |logit change| vs frozen LM at init: 0
  step 0 grads: |d/d alpha_xattn| per layer 0.0005, 0.000685; largest grad of any other new weight 0
  step    0 loss 0.362  tanh gates (xattn/dense) layer1 +0.00/+0.00  layer2 +0.00/+0.00
  step  300 loss 0.091  tanh gates (xattn/dense) layer1 +0.09/+0.10  layer2 +0.10/+0.20
  step  600 loss 0.037  tanh gates (xattn/dense) layer1 +0.09/+0.11  layer2 +0.08/+0.28
  held-out accuracy     colour 1.00  shape 0.99  side 1.00

[ungated, no spatial pos] max |logit change| vs frozen LM at init: 1.98
  held-out accuracy     colour 1.00  shape 0.98  side 1.00

frozen LM weights unchanged after all training: True
image 1 changed -> on chunk 2, cross-attention output changes by 0 (layer 1), 8.73 (layer 2); logits by 0.818
no visible image, trained model vs frozen LM: max |logit change| 5.91

frames  visual features  resampler output  scores per text token (resampled / all features)
     1               16           (8, 64)             8 / 16
     4               64           (8, 64)             8 / 64
     8              128           (8, 64)             8 / 128
    30              480           (8, 64)             8 / 480

The first two checks are exact, not statistical. With both gates at zero the model’s logits equal the frozen language model’s to the last bit, while the ungated model changes them by up to 1.98 from the first step. At step 0 only the gate scalars received gradient: every cross-attention, feed-forward and resampler weight had a gradient of exactly zero, so training begins by opening the gates. Over training the gates opened, but stayed small, with $|\tanh \alpha|$ between 0.08 and 0.28 by step 600, the dense gate of the second layer largest; their sign is arbitrary, because the layers behind them can flip the sign of their own output. The language model’s weights were bit-for-bit unchanged after all three trainings.

The frozen language model alone scores at chance on the attributes (about 1/3 for colour and shape, 1/2 for side), as it must. With cross-attention, colour and side reached 1.00 in every variant. Side was recovered even without spatial position embeddings, although attention over the unordered set of visual features cannot tell where a feature came from; in our reading, the convolutions’ zero padding leaves position in the feature channels, the kind of implicit position the authors rely on for NFNet. Shape was harder: 0.83 without spatial embeddings against 0.99 with them and 0.98 for the ungated model. These differences between variants come from one seed and one short schedule, and we did not run others; they are not evidence about the paper’s ablations.

The masking check alters only the first image of each sequence. On the tokens of the second chunk, the first cross-attention layer’s output did not change at all: those tokens can only see the second image. The second layer’s output did change, because its queries come from hidden states that the frozen self-attention of the first language-model block had already mixed with the first chunk, and so did the final logits. This is the paper’s point that earlier images still reach a token through self-attention. The last check shows what the gate does not guarantee: after training, a sequence with no visible image no longer gets the frozen model’s output, because the gated feed-forward layer acts on every token whether or not an image precedes it. The equality holds at initialization only, which is all the paper claims. The resampler table shows the fixed token count: from 1 to 30 frames the output stays 8 tokens, so a text token always computes 8 cross-attention scores per head instead of 16 to 480; the 30-frame case uses linearly interpolated temporal embeddings, as Flamingo does when it moves from 8 training frames to 30.

What the toy cannot show: in-context few-shot learning, which in the paper emerges from billions of parameters and web-scale interleaved data; the value of contrastive pretraining for the encoder (ours is random); the paper’s ablation results, such as the 4.2-point loss without gating or the instability, since our ungated model trained as well as the gated ones; and anything about video quality, since the video resampler is untrained and only its shapes are checked. Our language model also saw <image> and <EOC> tokens during its text-only pretraining, whereas Flamingo adds <EOC> to the vocabulary and learns it.

Inference

A task is posed as text completion with an interleaved prompt: a few support examples, each an image or video followed by its text, then the query image or video (§2.5, Appendix A.2). Two templates cover almost every benchmark: “Output: {output}” for captioning and other vision-to-text tasks, and “Question: {question} Answer: {answer}” for visual question answering and dialogue. HatefulMemes gets a template that includes the meme’s OCR text, the only benchmark where OCR text is given as input; TextVQA relies on the model’s own reading (Appendix B.1.5). The support examples are concatenated in random order unless stated otherwise (Appendix A.2).

• Open-ended tasks are decoded with beam search, beam size 3, stopping at the first predicted <EOC> (Appendix A.2).
• Close-ended tasks, such as multiple choice and classification, append each candidate answer to the query and rank the candidates by the model’s log-likelihood (Appendix A.2).
• Zero-shot results use a prompt with two text-only examples from the task, images removed, to show the output format; one example biased the model towards copying it, and more than two helped only marginally (Appendix A.2).
• RICES (Retrieval-based In-Context Example Selection), used for large support sets, retrieves the support images most similar to the query by the frozen encoder’s features and orders them by increasing similarity, so the most similar one sits next to the query. Prompt ensembling averages log-likelihoods over 6 permutations of the selected examples (Appendix A.2).

Evaluation hyperparameters, prompts included, were fixed across benchmarks and tuned jointly on the validation subsets of the development benchmarks, because tuning a prompt per task on its examples amounts, in the authors’ words, to a basic few-shot learning method that uses extra labelled data (Appendix B.1.5). In-context learning needs no gradient updates, but its cost grows with the number of shots (Appendix D.1), and the prompt length is bounded by the 2,048-token sequence length the language models were trained on: a 32-shot VisDial prompt would run to 4,096–8,192 tokens, so VisDial results are capped at 16 shots (Appendix D.1).

Performance

Few-shot results

The paper evaluates on 16 image and video benchmarks. Five of them, COCO, OKVQA, VQAv2, MSVDQA and VATEX, formed the development set used for design decisions, so their estimates may be biased; the other 11 were held out and used only to estimate few-shot performance (§3). For each benchmark the query set is the split prior work reports on, and support examples come from disjoint subsets of the training set; question-answering benchmarks are evaluated on 1,024 query samples, COCO captioning on 5,000 and VATEX on 1,024 (Appendix B.1.4).

Benchmark	Split, metric	Previous zero/few-shot best (shots)	Flamingo-80B, 0 / 4 / 32 shots	Fine-tuned best (training examples)
OKVQA*	val, VQA accuracy	43.3 (16)	50.6 / 57.4 / 57.8	54.4 (10K)
VQAv2*	test-dev, VQA accuracy	38.2 (4)	56.3 / 63.1 / 67.6	80.2 (444K)
COCO*	test, CIDEr	32.2 (0)	84.3 / 103.2 / 113.8	143.3 (500K)
MSVDQA* (video)	test, top-1 accuracy	35.2 (0)	35.6 / 41.7 / 52.3	47.9 (27K)
VATEX* (video)	test, CIDEr	—	46.7 / 56.0 / 65.1	76.3 (500K)
VizWiz	test-dev, VQA accuracy	—	31.6 / 39.6 / 49.8	57.2 (20K)
Flickr30K	test (Karpathy), CIDEr	—	67.2 / 75.1 / 75.4	67.4 (30K)
MSRVTTQA (video)	test, top-1 accuracy	19.2 (0)	17.4 / 23.9 / 31.0	46.8 (130K)
iVQA (video)	test, iVQA accuracy	12.2 (0)	40.7 / 44.1 / 45.3	35.4 (6K)
YouCook2 (video)	val, CIDEr	—	60.1 / 74.5 / 86.8	138.7 (10K)
STAR (video)	test, top-1 accuracy (multiple choice)	39.4 (0)	39.7 / 42.4 / 42.2	36.7 (46K)
VisDial	val, NDCG	11.6 (0)	52.0 / 55.6 / 55.6	75.2 (123K)
TextVQA	val, VQA accuracy	—	35.0 / 36.5 / 37.9	54.7 (20K)
NextQA (video)	test, WUPS	—	26.7 / 30.8 / 33.5	25.2 (38K)
HatefulMemes	test seen, ROC AUC	66.1 (0)	46.4 / 68.6 / 70.0	79.1 (9K)
RareAct (video)	test, mWAP	40.7 (0)	60.8 (zero-shot only)	—

Values from Table 1 (Flamingo-80B rows and comparison rows), splits and metrics from Appendix Table 6; * marks the development benchmarks. Appendix D.1 says the VisDial results are capped at 16 shots, so the value in its 32-shot column presumably comes from at most 16. Table 1 labels NextQA an image benchmark, while Table 6 lists it under video. The VQA, COCO and Flickr30k dataset articles describe those benchmarks.

With 32 shots, Flamingo-80B beat the fine-tuned state of the art in Table 1 on six benchmarks: OKVQA, MSVDQA, Flickr30K, iVQA, STAR and NextQA. The text (§1, §3.1) and the caption of Figure 2 say 6 of 16, but the caption of Table 1 says seven; the table’s underlined best-overall entries number seven only if RareAct, which has no fine-tuned result, is counted, and STAR’s best entry there is Flamingo-9B at 4 shots (42.8). Performance grew with model size and with the number of shots, and the largest model gained most from more shots (§3.1, Figure 2). Zero-shot HatefulMemes, at 46.4 ROC AUC, is below the 50 of chance.

Fine-tuned results

Fine-tuning was not the paper’s focus, but on the nine benchmarks where few-shot Flamingo-80B was not state of the art the authors fine-tuned it, keeping the language model frozen, unfreezing the vision encoder, raising the resolution to $480 \times 480$ and choosing learning rate, schedule, steps, batch size (8 or 16) and augmentation per task by grid search (§3.2, Appendix B.2.2).

Benchmark	Split	32-shot	Fine-tuned Flamingo-80B	Previous best
VQAv2	test-dev / test-std	67.6 / —	82.0 / 82.1	81.3† / 81.3†
COCO	test (CIDEr)	113.8	138.1	149.6†
VATEX	test (CIDEr)	65.1	84.2	81.4†
VizWiz	test-dev / test-std	49.8 / —	65.7 / 65.4	57.2† / 60.6†
MSRVTTQA	test	31.0	47.4	46.8
VisDial	valid / test-std (NDCG)	56.8 / —	61.8 / 59.7	75.2 / 75.4†
YouCook2	valid (CIDEr)	86.8	118.6	138.7
TextVQA	valid / test-std	36.0 / —	57.1 / 54.1	54.7 / 73.7
HatefulMemes	test seen (ROC AUC)	70.0	86.6	84.6†

Values from Table 2; † marks prior methods that use model ensembling or optimize the test metric directly (such as CIDEr optimization). The 32-shot VisDial and TextVQA entries in Table 2 (56.8 and 36.0) differ from Table 1’s 55.6 and 37.9; the table does not say why. Fine-tuned Flamingo set a new state of the art on VQAv2, VATEX, VizWiz, MSRVTTQA and HatefulMemes (§3.2). The appendix text gives the 32-shot VQAv2 result as 67.3%, against 67.6 in Tables 1 and 2 (Appendix B.2.2). The authors attribute some remaining gaps to optimizing only log-likelihood, without CIDEr optimization for captioning or dense-annotation fine-tuning for VisDial (Appendix B.2.2).

Classification and the contrastive encoder

Flamingo can classify by scoring class names as close-ended answers. On ImageNet, Flamingo-80B reached 66.4% top-1 with 16 random support examples, 76.0% when RICES chose the 16 from 5,000 (5 per class) and 77.3% with prompt ensembling added; on Kinetics700 the corresponding averages of top-1 and top-5 were 51.2, 63.5 and 64.2 (Appendix B.2.1, Table 7). The text of B.2.1 calls the RICES gain 9.2%; the table’s numbers differ by 9.6 points. All of these trail the zero-shot accuracy of Flamingo’s own contrastive encoder on ImageNet, 77.9%, and the contrastive state of the art, 85.7% (Table 7). That encoder, evaluated zero-shot on retrieval, reached 89.3 image-to-text and 79.5 text-to-image R@1 on Flickr30K and 65.9 and 48.0 on COCO (Appendix B.2.3, Table 9). These R@1 values are above CLIP’s and ALIGN’s in the table and above Florence’s on three of the four measures; the text says the encoder outperforms all three, but Table 9 gives Florence 90.9 for Flickr30K image-to-text R@1; the Image–Text Retrieval article explains the measures.

Strengths

• One model, many tasks, no gradient steps. A single set of weights handles captioning, open and multiple-choice question answering, dialogue and classification on images and video, adapted only by its prompt.
• Pretrained knowledge kept. Freezing both pretrained networks preserves what they learned; in the ablations, freezing beat both fine-tuning and co-training the language model, and it saves the language model’s optimizer state and gradients (Appendix B.3.1).
• Variable visual input at fixed cost. The resampler gives each image or clip 64 tokens whatever its resolution or length, and the per-image mask lets models trained on 5 images per sequence use 32 at test time.
• Interleaved data. Training on web pages, not only captions, is what the ablations tie most strongly to few-shot ability (removing M3W cost 17.3 points).

Limitations

The authors list the following (§5, Appendix D.1).

• Inherited language-model weaknesses. Flamingo inherits its language model’s priors, which the authors suspect contribute to occasional hallucinations and ungrounded guesses in open-ended dialogue (Figure 13); the Multimodal Hallucination article covers the problem. Causal modelling of the prompt is less expressive than bidirectional modelling, and the language models generalize poorly beyond their 2,048-token training length.
• Classification. Flamingo trails contrastive models on classification, including its own vision encoder, which the authors attribute to contrastive training optimizing retrieval directly.
• In-context learning’s own costs. Results are sensitive to the demonstrations and their order, and compute and performance scale poorly with the number of shots beyond the low-data regime; performance plateaus beyond 32 shots, whereas gradient-based methods keep gaining from more labelled data.
• Evaluation caveats. The development benchmarks guided design decisions. The image–text pairs were deduplicated against most benchmarks, but not against VizWiz, HatefulMemes and TextVQA, which were evaluated after training, and M3W was not deduplicated; a check of its 185 million images found 1,314 candidate duplicates of validation and test images, 125 of them exact (Appendix A.3.4).
• Scope. The model card lists the primary use as research, says the model is trained on English data, and advises against downstream use without application-specific safety and fairness mitigations (Appendix E). The paper also discusses risks of gender and racial bias and of toxic outputs (§5, Appendix D.2).
• Not reproducible from public resources. The OBELICS authors note that the datasets used to train Flamingo-style models had not been released (Laurençon et al., 2023), and Hugging Face’s IDEFICS model card describes Flamingo as closed-source.

Historical Importance

Flamingo showed that a frozen large language model and a frozen vision encoder, joined by a small fraction of new trainable parameters and trained on interleaved web data, could do few-shot in-context learning on images and video, and could match or beat fine-tuned specialists on several benchmarks with 32 examples. The authors describe it as the first language model that ingests arbitrarily interleaved images, videos and text (§4). Its components became standard parts of the vision–language model vocabulary: a resampler of learned queries as a connector, and gated cross-attention inside a frozen language model as an alternative to placing visual tokens in the input sequence, as the Vision–Language Model and Multimodal Fusion articles describe. BLIP-2 compared its connector, the Q-Former, with Flamingo’s resampler and its results with Flamingo-80B’s, as the BLIP article describes. The lack of a public interleaved dataset led to OBELICS and the open reproductions below.

Variants

The paper’s three sizes, Flamingo-3B, Flamingo-9B and Flamingo-80B, are its only variants; all share the vision encoder and resampler (§2.2).

OpenFlamingo is an open-source PyTorch reimplementation by Anas Awadalla, Irena Gao and colleagues, announced in early 2023; the v2 release of June 2023 provides five models at the 3B, 4B and 9B scales, built on OpenAI’s CLIP ViT-L/14 and MosaicML’s MPT-1B and MPT-7B or Together’s RedPajama-3B language models, with cross-attention every 1, 2 or 4 layers, trained on LAION-2B and Multimodal C4 (LAION blog, 2023; repository README). The repository’s README lists, for example, 89.0 CIDEr on COCO and 54.8% on VQAv2 with 4 shots for the 9B model. The blog reports that, averaged over 7 datasets and 0 to 32 shots, the models reach more than 80% of the corresponding Flamingo model’s performance, and trail it by about 10% at 0 shots and 15% at 32 shots; its introduction says the 3B and 9B models reach more than 60% of fine-tuned state of the art with 32 shots, but its results section says more than 55%. The code follows Flamingo closely: gates attn_gate and ff_gate initialized to 0 and applied through tanh, a 6-layer resampler with 64 latents whose keys and values include the latents, and a mask that restricts each text token to the most recent image. One default differs: the language model’s input embeddings are trained unless a freeze_lm_embeddings option is set (factory.py at the pinned commit). As of November 2026 the README still lists video input as a future plan. The team also describes the framework in a technical report (Awadalla et al., 2023).

IDEFICS (Image-aware Decoder Enhanced à la Flamingo with Interleaved Cross-attentionS) is Hugging Face’s open-access reproduction, released in 2023 at 80B and 9B parameters (Laurençon et al., 2023; model card). It combines a frozen LLaMA language model (65B and 7B) with a frozen OpenCLIP ViT-H/14 encoder, a 6-layer Perceiver Resampler with 64 latents, 16 heads and head dimension 96, and new cross-attention blocks every fourth layer. Layer normalization on the projected queries and keys of the resampler and cross-attention blocks improved training stability in the authors’ early experiments. It trained for 200K steps on OBELICS (73.85% of tokens), Wikipedia documents (6.15%), LAION pairs (17.18%) and PMD pairs (2.82%), with no video (model card). OBELICS, built from Common Crawl, holds 141 million web documents with 353 million images, 298 million of them unique, and 115 billion text tokens (Laurençon et al., 2023). With 32 random in-context examples, IDEFICS-80B matched or beat Flamingo-80B on 4 of 8 benchmarks, for example 116.6 against 113.8 CIDEr on COCO and 57.8 for both on OKVQA, while trailing on VQAv2 test-dev (65.9 against 67.6); the model card cautions that COCO is not directly comparable, because PMD contains COCO, and that Flamingo saw $320 \times 320$ images where IDEFICS and OpenFlamingo saw $224 \times 224$. IDEFICS-9B trailed Flamingo-9B but, by best score over shots, edged OpenFlamingo-9B in aggregate (56.5 against 55.8) (OBELICS, Table 2 and §5). The instruction-tuned versions, IDEFICS-instruct, unfreeze all parameters. The OBELICS table and the model card differ slightly in places, for example IDEFICS-80B’s 16-shot HatefulMemes score (57.8 against 55.8), and OBELICS lists Flamingo-9B’s zero-shot VizWiz as 22.8, where the Flamingo paper has 28.8.

Implementations

DeepMind’s implementation used JAX and Haiku (Appendix B.1.2); the paper gives pseudo-code for the resampler (Figure 5) and the GATED XATTN-DENSE block (Figure 4). The open reproductions are OpenFlamingo (PyTorch, GitHub mlfoundations/open_flamingo, with checkpoints on Hugging Face) and IDEFICS (Hugging Face, HuggingFaceM4). The IDEFICS model card releases the newly trained weights under the MIT licence, while the LLaMA weights they depend on carry a non-commercial research licence (as of November 2026).