Vision–Language Model

A vision–language model (VLM) takes an image, usually together with a text prompt, and generates text with a language model. The current form, also called a multimodal large language model (MLLM), joins three parts: a pretrained vision encoder, typically a CLIP-style Vision Transformer; a connector that turns image features into something the language model can consume; and a pretrained large language model (LLM), a decoder-only transformer. Captioning, visual question answering and document reading become instructions to one model. Multimodal Learning places VLMs in the wider field, and Vision–Language Pretraining covers the encoder-style models that preceded them.

Core Idea

A language model predicts text from a sequence of token embeddings. If an image can be expressed as vectors in that space, or injected into its hidden states, the model can condition on it without being redesigned. A VLM therefore reuses a vision encoder trained on image–text pairs and an LLM trained on text, trains a comparatively small bridge between them, and then adapts the LLM with image–text instructions.

Building Blocks

Vision encoder. Most VLMs use a ViT pretrained contrastively on image–text pairs, whose features already align with language: CLIP ViT-L/14 in LLaVA (Liu et al., 2023), EVA-CLIP’s ViT-g/14 in BLIP-2 (Li et al., 2023), OpenCLIP’s ViT-bigG in Qwen-VL (Bai et al., 2023). Flamingo pretrained an NFNet-F6 with a CLIP-style loss (Alayrac et al., 2022). The encoder is often frozen, but not always: Qwen-VL trains it during pretraining, and the Llama 3 team found that updating theirs helped, especially for text recognition (Llama Team, 2024).

Connector. Three designs recur:

• A projector: a linear layer (LLaVA) or two-layer MLP (LLaVA-1.5) applied to every patch feature, so each patch becomes one LLM token (Liu et al., 2023; Liu et al., 2024).
• A query-based resampler: learned queries, like DETR’s object queries, cross-attend to the patch features and return a fixed number of tokens regardless of image size. BLIP-2’s Q-Former (32 queries) is described in the vision–language pretraining article; Flamingo’s Perceiver Resampler returns 64 tokens (Alayrac et al., 2022); Qwen-VL’s single cross-attention layer returns 256 and adds 2D absolute position encodings to queries and keys to keep location (Bai et al., 2023).
• Gated cross-attention layers inserted into the LLM, so that text tokens attend to visual features at several depths (Flamingo); see below and Multimodal Fusion.

Language model. A pretrained LLM generates the output and dominates the parameter count: Qwen-VL’s 9.6B parameters split into 1.9B for the vision encoder, 0.08B for the adapter and 7.7B for the Qwen-7B LLM (Bai et al., 2023).

How Data Flows

Visual tokens in the input sequence. The connector’s outputs are concatenated with the embedded text, and the LLM’s causal self-attention does the fusion in every layer, as in LLaVA and BLIP-2 (Li et al., 2023). The LLM needs no change, but every visual token lengthens the sequence.

Cross-attention inside the language model. The sequence stays text-only, and new cross-attention layers let text tokens read visual features. Flamingo inserts a gated cross-attention block before every seventh frozen Chinchilla layer in Flamingo-80B and masks the cross-attention so each text token sees only the image that most recently preceded it, which lets a prompt interleave several images with text (Alayrac et al., 2022). The Llama 3 team took the same route for their image adapter, with cross-attention after every fourth self-attention layer and the language model’s weights left untouched; for the 405B model the adapter alone has about 100B parameters (Llama Team, 2024). This keeps the sequence short and text-only behavior intact, at the cost of new parameters.

Example: counting visual tokens and their cost

The script follows one $336 \times 336$ synthetic image through a LLaVA-1.5-like pipeline: $14 \times 14$ patches give 576 tokens of width 1,024 (a linear patch embedding stands in for the ViT, whose blocks keep this shape); a two-layer MLP projects them to an illustrative LLM width of 4,096; and they join a 40-token prompt. Per-block cost uses the transformer article’s estimate of $12nd^2 + 2n^2d$ multiply-adds. It then swaps in 32 learned queries and counts tokens for the high-resolution schemes under Variants. Weights are random; only shapes and costs matter. It runs in under a second with NumPy 2.3.

import numpy as np

rng = np.random.default_rng(0)
P, D_VIS, D_LLM, D_Q = 14, 1024, 4096, 768   # ViT-L/14 patch and width; illustrative LLM width

def image(h, w):
    """A synthetic RGB image: a red disc on a grey background, values in [0, 1]."""
    yy, xx = np.mgrid[0:h, 0:w]
    img = np.full((h, w, 3), 0.5, dtype=np.float32)
    img[(yy - h / 2) ** 2 + (xx - w / 2) ** 2 < (min(h, w) / 4) ** 2] = (0.9, 0.1, 0.1)
    return img

def patchify(img, p=P):
    """(H, W, 3) -> (H/p * W/p, p*p*3): one row per non-overlapping p x p patch."""
    h, w, c = img.shape
    x = img.reshape(h // p, p, w // p, p, c).transpose(0, 2, 1, 3, 4)
    return x.reshape(-1, p * p * c)

def gelu(x):
    return 0.5 * x * (1 + np.tanh(0.7978845608 * (x + 0.044715 * x ** 3)))

def block_cost(n, d):
    """Multiply-adds of one transformer block on n tokens of width d:
    about 12 n d^2 in the linear layers and 2 n^2 d in QK^T and the weighted sum."""
    return 12 * n * d * d + 2 * n * n * d

# 1. Vision encoder (stand-in): patchify + linear patch embedding. A real ViT-L/14
#    then runs 24 transformer blocks; they keep the shape (576, 1024).
patches = patchify(image(336, 336))
W_pe = rng.standard_normal((patches.shape[1], D_VIS), dtype=np.float32) / np.sqrt(patches.shape[1])
Z = patches @ W_pe
print(f"image 336x336x3 -> patches {patches.shape} -> visual features Z {Z.shape}")

# 2. Connector: a two-layer MLP projector into the LLM's embedding width.
W1 = rng.standard_normal((D_VIS, D_LLM), dtype=np.float32) / np.sqrt(D_VIS)
W2 = rng.standard_normal((D_LLM, D_LLM), dtype=np.float32) / np.sqrt(D_LLM)
H_v = gelu(Z @ W1) @ W2
print(f"MLP projector {D_VIS}->{D_LLM}->{D_LLM}: H_v {H_v.shape}, "
      f"{(W1.size + W2.size) / 1e6:.1f}M weights")

# 3. Concatenate with the embedded text prompt (40 tokens of an invented prompt).
n_text = 40
H_t = rng.standard_normal((n_text, D_LLM), dtype=np.float32)
seq = np.concatenate([H_v, H_t])
print(f"LLM input: {H_v.shape[0]} visual + {n_text} text = {seq.shape} tokens")
for name, n in [("text only", n_text), ("image + text", seq.shape[0])]:
    print(f"  {name:13s} n = {n:4d}  one block: {block_cost(n, D_LLM) / 1e9:6.1f} G multiply-adds, "
          f"attention scores per head: {n * n:,}")
print(f"  cost ratio image+text / text: {block_cost(seq.shape[0], D_LLM) / block_cost(n_text, D_LLM):.1f}x")

# 4. A Q-Former-like resampler: 32 learned queries cross-attend to the 576 features.
Qry = rng.standard_normal((32, D_Q), dtype=np.float32)
Wq, Wk, Wv = (rng.standard_normal(s, dtype=np.float32) / np.sqrt(s[0])
              for s in [(D_Q, D_Q), (D_VIS, D_Q), (D_VIS, D_Q)])
S = (Qry @ Wq) @ (Z @ Wk).T / np.sqrt(D_Q)            # (32, 576) attention scores
A = np.exp(S - S.max(1, keepdims=True)); A /= A.sum(1, keepdims=True)
out = A @ (Z @ Wv)                                     # (32, 768)
H_q = out @ (rng.standard_normal((D_Q, D_LLM), dtype=np.float32) / np.sqrt(D_Q))
xattn = 32 * D_Q * D_Q + 2 * 576 * D_VIS * D_Q + 2 * 32 * 576 * D_Q
print(f"\nQ-Former-like: scores {S.shape} -> queries {out.shape} -> LLM tokens {H_q.shape}")
print(f"  one cross-attention layer: {xattn / 1e9:.2f} G multiply-adds")
n = 32 + n_text
print(f"  LLM input n = {n}: one block {block_cost(n, D_LLM) / 1e9:.1f} G, "
      f"{block_cost(seq.shape[0], D_LLM) / block_cost(n, D_LLM):.1f}x cheaper than with 576 visual tokens")

# 5. Visual token budgets of high-resolution strategies.
def grid_tokens(h, w, p=P):
    return (h // p) * (w // p)

print("\nvisual tokens per image")
print(f"  single 336x336 view:                   {grid_tokens(336, 336):5d}")
print(f"  2x2 tiles of 336 + 336 global view:    {4 * grid_tokens(336, 336) + grid_tokens(336, 336):5d}")
print(f"  2x2 tiles of 224 + 224 global view:    {4 * grid_tokens(224, 224) + grid_tokens(224, 224):5d}")
for h, w in [(224, 224), (448, 672)]:
    print(f"  native {h}x{w}, 2x2 token merge:      {grid_tokens(h, w) // 4:5d}  (+2 start/end tokens)")

image 336x336x3 -> patches (576, 588) -> visual features Z (576, 1024)
MLP projector 1024->4096->4096: H_v (576, 4096), 21.0M weights
LLM input: 576 visual + 40 text = (616, 4096) tokens
  text only     n =   40  one block:    8.1 G multiply-adds, attention scores per head: 1,600
  image + text  n =  616  one block:  127.1 G multiply-adds, attention scores per head: 379,456
  cost ratio image+text / text: 15.8x

Q-Former-like: scores (32, 576) -> queries (32, 768) -> LLM tokens (32, 4096)
  one cross-attention layer: 0.95 G multiply-adds
  LLM input n = 72: one block 14.5 G, 8.7x cheaper than with 576 visual tokens

visual tokens per image
  single 336x336 view:                     576
  2x2 tiles of 336 + 336 global view:     2880
  2x2 tiles of 224 + 224 global view:     1280
  native 224x224, 2x2 token merge:         64  (+2 start/end tokens)
  native 448x672, 2x2 token merge:        384  (+2 start/end tokens)

The image makes each LLM block 15.8 times more expensive than the prompt alone; 32 resampled tokens make it 8.7 times cheaper again. By our arithmetic, the linear layers still dominate at these lengths, so cost grows roughly linearly in tokens; the quadratic term takes over beyond $n = 6d$, about 24,600 tokens here, which many tiles or video frames can reach.

Mathematical Formulation

Let $x$ be the image and $y = (y_1, \dots, y_L)$ the text. A VLM models

$$p(y \mid x) = \prod_{\ell=1}^{L} p_\theta\big(y_\ell \mid y_{<\ell}, x\big)$$

and is trained with the negative log-likelihood of the target tokens; in instruction data, only the answer tokens count (Liu et al., 2023).

Input-sequence style. With encoder features $Z = f(x) \in \mathbb{R}^{n_v \times d_v}$ and connector $g_\phi$, the visual tokens are $H_v = g_\phi(Z) \in \mathbb{R}^{n_v \times d}$ ($H_v = Z W$ for LLaVA’s linear projector). The LLM reads $[H_v; E(y)]$, with $E$ its token embedding.

Resampler. With learned queries $Q \in \mathbb{R}^{m \times d_q}$,

$$R = \operatorname{softmax}\!\left(\frac{(Q W_q)(Z W_k)^\top}{\sqrt{d_q}}\right) Z W_v \in \mathbb{R}^{m \times d_q},$$

so the LLM receives $m$ tokens ($m = 32$ in BLIP-2, 64 in Flamingo, 256 in Qwen-VL) however large $n_v$ is. Qwen-VL uses one such layer; BLIP-2’s Q-Former and Flamingo’s resampler are transformers that stack several, with self-attention among the queries.

Gated cross-attention. For text hidden states $h$ and visual tokens $V$, a Flamingo block adds

$$h \leftarrow h + \tanh(\alpha_{\text{xattn}})\,\operatorname{XAttn}(h, V), \qquad h \leftarrow h + \tanh(\alpha_{\text{dense}})\,\operatorname{FFW}(h),$$

with scalars $\alpha$ initialized to 0, so training starts from the frozen LLM (Alayrac et al., 2022).

Why It Works

In our reading, two properties carry the design. A contrastive image encoder already arranges images by the concepts captions name, so mapping its features into an LLM’s embedding space is close to a change of coordinates; a linear map sufficed for LLaVA’s first stage, which its authors describe as training a visual tokenizer for the frozen LLM (Liu et al., 2023). And the LLM brings knowledge and fluency learned from far more text than any paired dataset, which lets a VLM use knowledge beyond the image but also write fluent answers the image does not support.

Training Considerations

Training typically proceeds in stages that unfreeze progressively more of the model.

1. Alignment pretraining. The connector is trained on image–caption pairs to generate the caption, with the encoder and LLM frozen. LLaVA trained only $W$ on 595K pairs filtered from Conceptual Captions; LLaVA-1.5 used 558K (Liu et al., 2023; Liu et al., 2024). Qwen-VL instead trained encoder and adapter on 1.4B web pairs with the LLM frozen, then the whole model at $448 \times 448$ (Bai et al., 2023).
2. Visual instruction tuning. The connector and LLM are fine-tuned on (image, instruction, answer) examples. LLaVA used 158K examples that the text-only GPT-4 wrote from the captions and bounding boxes of COCO images (the conversation prompt uses the captions only): 58K conversations, 23K detailed descriptions and 77K complex-reasoning answers (Liu et al., 2023). InstructBLIP converted 26 public datasets to instructions, tuned on 13, evaluated zero-shot on the other 13, and fed the instruction to the Q-Former too, so the visual features depend on the question (Dai et al., 2023).
3. Preference tuning. DPO fits pairs of preferred and rejected answers with a classification-style loss instead of fitting a reward model and running reinforcement learning (Rafailov et al., 2023). RLHF-V collected segment-level human corrections of hallucinated passages and trained with a dense variant of DPO; with 1.4K annotated samples it reduced its base model’s object hallucination on the authors’ MHumanEval benchmark by 34.8%, a relative drop (Yu et al., 2024).

Freezing trades capacity for cost and stability: a frozen LLM keeps its text abilities and needs no optimizer state, but LLaVA-1.5’s authors attribute part of InstructBLIP’s difficulty balancing short and long answers to tuning only the Q-Former (Liu et al., 2024).

Example: a tiny VLM on shape images

Each $32 \times 32$ image shows one red, green or blue circle, square or triangle in the left or right half; each question is one of three invented templates with a one-word answer. The frozen vision encoder is two random, never-trained convolution layers giving a $4 \times 4$ grid of 64-dimensional features; a linear projector maps them to 16 visual tokens; and a two-layer causal transformer, trained from scratch, reads them and the question and predicts the answer. Without visual tokens it is a text-only baseline. It uses PyTorch 2.14.1 and runs in about 10 seconds.

import sys
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(int(sys.argv[1]) if len(sys.argv) > 1 else 0)
torch.set_num_threads(4)
S = 32
COLORS = {"red": (1., .15, .15), "green": (.15, 1., .15), "blue": (.15, .15, 1.)}
SHAPES = ["circle", "square", "triangle"]
SIDES = ["left", "right"]
QUESTIONS = ["what color is it ?", "what shape is it ?", "what side is it ?"]   # invented
VOCAB = sorted({w for q in QUESTIONS for w in q.split()}) + list(COLORS) + SHAPES + SIDES
IDX = {w: i for i, w in enumerate(VOCAB)}
yy, xx = torch.meshgrid(torch.arange(S) + .5, torch.arange(S) + .5, indexing="ij")

def draw(color, shape, side):
    """A 32x32 RGB image with one colored shape in the left or right half."""
    cx = (8. if side == "left" else 24.) + torch.empty(1).uniform_(-3, 3)
    cy = 16. + torch.empty(1).uniform_(-6, 6)
    r = torch.empty(1).uniform_(4, 6)
    x, y = xx - cx, yy - cy
    if shape == "circle":
        m = x ** 2 + y ** 2 < r ** 2
    elif shape == "square":
        m = (x.abs() < .85 * r) & (y.abs() < .85 * r)
    else:
        m = (y < .8 * r) & (y > -r + 2 * x.abs())
    img = .1 + m[None] * torch.tensor(COLORS[color])[:, None, None]
    return img + .05 * torch.randn(3, S, S)

def make_data(n):
    imgs, qs, ans = [], [], []
    for _ in range(n):
        c = list(COLORS)[torch.randint(3, (1,))]
        s = SHAPES[torch.randint(3, (1,))]
        d = SIDES[torch.randint(2, (1,))]
        k = int(torch.randint(3, (1,)))
        imgs.append(draw(c, s, d))
        qs.append(torch.tensor([IDX[w] for w in QUESTIONS[k].split()]))
        ans.append(IDX[(c, s, d)[k]])
    return torch.stack(imgs), torch.stack(qs), torch.tensor(ans)

# Frozen "vision encoder": random convolutions, never trained -> 4x4 grid of 64-d tokens.
encoder = nn.Sequential(nn.Conv2d(3, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
                        nn.Conv2d(32, 64, 3, padding=1), nn.ReLU(), nn.MaxPool2d(4))
encoder.requires_grad_(False)

class TinyVLM(nn.Module):
    def __init__(self, n_vis, d=64):
        super().__init__()
        self.n_vis = n_vis                                 # 0: text only
        self.proj = nn.Linear(64, d)                       # connector: linear projector
        self.embed = nn.Embedding(len(VOCAB), d)
        self.pos = nn.Parameter(torch.randn(n_vis + 5, d) * .02)
        layer = nn.TransformerEncoderLayer(d, 4, 4 * d, dropout=0., batch_first=True, norm_first=True)
        self.decoder = nn.TransformerEncoder(layer, 2, enable_nested_tensor=False)
        self.head = nn.Linear(d, len(VOCAB))

    def forward(self, feats, q):
        x = self.embed(q)
        if self.n_vis:
            x = torch.cat([self.proj(feats), x], 1)         # visual tokens prepended
        x = x + self.pos[: x.shape[1]]
        mask = nn.Transformer.generate_square_subsequent_mask(x.shape[1])
        h = self.decoder(x, mask=mask, is_causal=True)     # causal mask: a decoder
        return self.head(h[:, -1])                         # the word after "?"

tr, te = make_data(3000), make_data(1000)
with torch.no_grad():                                      # frozen: features computed once
    f_tr, f_te = (encoder(x).flatten(2).transpose(1, 2) for x in (tr[0], te[0]))   # (N, 16, 64)
kind = te[1][:, 1]                                         # "color", "shape" or "side"
for name, n_vis in [("with visual tokens", 16), ("text only", 0)]:
    model = TinyVLM(n_vis)
    opt = torch.optim.Adam(model.parameters(), lr=3e-3)
    for step in range(600):
        i = torch.randint(len(tr[0]), (128,))
        loss = F.cross_entropy(model(f_tr[i], tr[1][i]), tr[2][i])
        opt.zero_grad(); loss.backward(); opt.step()
    with torch.no_grad():
        ok = model(f_te, te[1]).argmax(1) == te[2]
    per = "  ".join(f"{w} {ok[kind == IDX[w]].float().mean():.2f}" for w in ("color", "shape", "side"))
    print(f"{name:18s} accuracy {ok.float().mean():.3f}   {per}")

with visual tokens accuracy 0.932   color 1.00  shape 0.78  side 1.00
text only          accuracy 0.407   color 0.35  shape 0.33  side 0.53

Without the image, the model learns which words answer each question but must guess among them (chance about 0.39 overall). With visual tokens it answers color and side almost perfectly; shape is harder, because the frozen random features, pooled to a $4 \times 4$ grid, keep little of an outline. Over seeds 0–9, accuracy with visual tokens ranged from 0.919 to 0.967 (shape 0.77–0.91) and without them from 0.364 to 0.416.

Variants

High-resolution input. Encoders pretrained at $224$ or $336$ pixels miss small text. LLaVA-1.5-HD splits the image into up to six $224 \times 224$ tiles (grids such as $1 \times 3$, $2 \times 2$ or $2 \times 3$), encodes each separately, merges the feature maps, appends a special token to each row to mark the layout, and adds features of a downsampled copy for global context (Liu et al., 2024). A $2 \times 2$ grid plus the global view gives 1,280 tokens in the walk-through, before row-end tokens. Qwen2-VL instead removes the ViT’s absolute position embeddings in favor of 2D rotary embeddings (see positional encoding), so the encoder accepts images at their native resolution; an MLP merges each $2 \times 2$ group of adjacent tokens, so a $224 \times 224$ image becomes 66 tokens including start and end markers. Its M-RoPE splits the LLM’s rotary position into temporal, height and width components (Wang et al., 2024).

Encoder–decoder language models. PaLI feeds all patch tokens of a 4B-parameter ViT-e, without pooling, into the encoder of an mT5-XXL encoder–decoder; for PaLI-17B the authors built WebLI, 10 billion images with text in over 100 languages (Chen et al., 2023).

Early fusion. Instead of attaching a pretrained encoder, Chameleon converts each $512 \times 512$ image into 1,024 discrete tokens from an 8,192-entry codebook and trains one transformer from scratch on interleaved image and text tokens, so it can also generate images (Chameleon Team, 2024).

Boxes as text. Qwen-VL writes bounding boxes as text, with coordinates normalized to $[0, 1000)$ between special tokens, so visual grounding needs no extra head (Bai et al., 2023).

Use in Computer Vision

VLMs are applied to captioning, visual question answering, document and chart reading, grounding, and multi-image or video dialogue. Of 16 image and video benchmarks, Flamingo set a new few-shot state of the art on the 9 with published few-shot results and beat fine-tuned state-of-the-art models on 6 (Alayrac et al., 2022).

Broad benchmarks complement single-task datasets:

• MMMU collects 11.5K college-level questions with images from six disciplines, 30 subjects and 183 subfields. GPT-4V scored 55.7% on the test set; the best open models the authors evaluated, BLIP-2 FLAN-T5-XXL and LLaVA-1.5-13B, about 34% (Yue et al., 2024).
• MMBench asks over 3,000 multiple-choice questions, each several times with shuffled choices, and counts it correct only if every attempt is (Liu et al., 2024, MMBench).
• POPE measures object hallucination by asking “Is there a ⟨object⟩ in the image?” about objects that are present and absent, with absent objects drawn at random, from the most frequent objects, or from those that most often co-occur with the true ones (Li et al., 2023, POPE).

Limitations

• Hallucination. VLMs describe objects that are not in the image, more often objects frequent in, or co-occurring with the image’s objects in, their instruction data (Li et al., 2023, POPE). LLaVA-1.5’s authors observed fewer hallucinations at $448 \times 448$ input and argue that models learn to hallucinate when training captions contain details their resolution cannot resolve (Liu et al., 2024).
• Perception before reasoning. Detail is lost when an image is squeezed into a few hundred tokens from a low-resolution encoder; of 150 GPT-4V errors on MMMU, 35% were perceptual, 29% came from missing knowledge and 26% from flawed reasoning (Yue et al., 2024). On the abilities where even the best model scored lowest, MMBench’s authors found all models weak at low-level properties such as brightness or sharpness, at tables and diagrams, and at spatial relations between objects (Liu et al., 2024, MMBench).
• Evaluation contamination and format sensitivity. Training mixtures include benchmarks’ training splits (LLaVA-1.5 marks such results), web data may contain test images, and scores depend on how answers are extracted from free text.
• Cost. Every visual token passes through every LLM layer, so tiling, multiple images and video multiply inference cost.

Important Models

Proprietary systems such as GPT-4V and Gemini also accept images; they are left out of the table.

Model	Year	Vision encoder	Connector	Language model	Visual tokens per image
Flamingo	2022	NFNet-F6, frozen	Perceiver Resampler + gated cross-attention	Chinchilla (1.4B–70B), frozen	64 (via cross-attention)
PaLI	2023	ViT-e (4B)	none: patch tokens into the text encoder	mT5 encoder–decoder	all patches
BLIP-2	2023	CLIP ViT-L/14 or EVA-CLIP ViT-g/14, frozen	Q-Former + linear layer	OPT or FlanT5, frozen	32
LLaVA	2023	CLIP ViT-L/14, frozen	linear projection	Vicuna	one per patch
InstructBLIP	2023	ViT-g/14, frozen	instruction-aware Q-Former	FlanT5 or Vicuna, frozen	32
Qwen-VL	2023	OpenCLIP ViT-bigG	one cross-attention layer, learned queries	Qwen-7B	256
IDEFICS	2023	OpenCLIP ViT, frozen	Perceiver blocks + cross-attention	LLaMA, frozen	64 (via cross-attention)
LLaVA-1.5	2024	CLIP ViT-L/14 at 336 px	two-layer MLP	Vicuna (7B, 13B)	576
Qwen2-VL	2024	675M ViT with 2D-RoPE	MLP merging 2×2 tokens	Qwen2 (1.5B–72B)	variable (66 at 224×224)

Years are those of the cited papers. LLaVA’s paper does not state its input resolution; LLaVA-1.5’s 576 is our arithmetic, $(336/14)^2$, and InstructBLIP’s 32 follows from its BLIP-2 initialization. Open reproductions of Flamingo include IDEFICS, an 80B-parameter model trained on OBELICS, 141 million web documents with 353 million images (Laurençon et al., 2023), and OpenFlamingo, whose 3B–9B models averaged 80–89% of the corresponding Flamingo models’ performance on seven datasets (Awadalla et al., 2023).