LLaVA

LLaVA (Large Language and Vision Assistant) is a family of open vision–language models that answer questions about images, describe them and converse about them. Haotian Liu, Chunyuan Li, Qingyang Wu and Yong Jae Lee introduced it in 2023 (NeurIPS) together with visual instruction tuning: fine-tuning a language model on (image, instruction, response) examples, which they generated by giving a text-only GPT-4 the captions and boxes of COCO images. It is perhaps the simplest form of the VLM design: a frozen CLIP Vision Transformer, a projector that turns each patch feature into a token, and a decoder-only transformer language model. LLaVA-1.5 (2024) made it a strong benchmark baseline; later versions added high-resolution tiling, multiple images and video.

Problem Addressed

By early 2023, machine-generated instruction data had turned LLaMA into assistants such as Alpaca and Vicuna, but multimodal models had not been instruction-tuned. Asked what is unusual about a photo, BLIP-2 and OpenFlamingo described the photo instead of answering (Liu et al., 2023). The obstacle was data: no large set of visual instructions with good responses existed, and crowdsourcing one is slow and ill-defined. LLaVA had GPT-4 write the data from existing annotations, and kept the model simple enough to iterate on that data quickly.

Architecture

A projection turns the vision encoder’s features into visual tokens, which the language model reads before the instruction; fusion happens in its causal self-attention.

Vision Encoder

The original model uses CLIP ViT-L/14, frozen throughout. Its paper does not state the input resolution; the LLaVA baseline in LLaVA-1.5’s ablation (Table 2) is listed at $224$ pixels, which would give a $16 \times 16$ grid of 256 patch features (our arithmetic). On ScienceQA, features from the second-to-last layer beat the last layer by 0.96 points; the authors hypothesize that the last layer focuses on global, abstract properties (Liu et al., 2023). LLaVA-1.5 moved to CLIP ViT-L/14 at $336$ pixels, the highest resolution available for CLIP, giving 576 tokens (our arithmetic, $(336/14)^2$) (Liu et al., 2024).

Projector

LLaVA uses one trainable matrix $W$: $H_v = W \cdot Z_v$ with $Z_v = g(X_v)$, so every patch becomes one token in the language model’s word-embedding space. The authors chose it because it is lightweight and makes data experiments fast, leaving Flamingo’s cross-attention and BLIP-2’s Q-Former as alternatives (Liu et al., 2023). LLaVA-1.5 replaced it with a two-layer MLP, citing the gains of MLP projection heads in self-supervised learning (Liu et al., 2024).

Language Model

The language model is Vicuna, an instruction-tuned LLaMA, chosen because it followed instructions best among public checkpoints. The ScienceQA results use the 13B model; a 7B model scored 1.08 points lower (Liu et al., 2023). LLaVA-1.5 uses Vicuna-v1.5 at 7B and 13B, the best of the four language models it compared (Liu et al., 2024).

Training

Instruction data from text-only GPT-4

Each COCO image was represented by its five captions and its labelled bounding boxes in normalized coordinates, although the conversation prompt printed in the paper’s appendix uses the captions alone. From this context and a few hand-written seed examples per type, the only human annotation in the pipeline, GPT-4 wrote conversations with questions that have definite answers (object types, counts, actions, locations, relative positions), detailed descriptions, and complex reasoning answers. LLaVA-Instruct-158K has 58K conversations, 23K detailed descriptions and 77K complex-reasoning samples on about 80K unique images; GPT-4 gave consistently better data than ChatGPT, for instance on spatial reasoning (Liu et al., 2023).

Example: instruction data from symbols, not pixels

The script builds a LLaVA-style context for an invented street scene and fills three template types from it; the templates stand in for GPT-4 and, unlike it, use only the boxes. It counts rough tokens (words and punctuation) and the share of each training sequence that is answer text, then draws the scene with a dog the annotators missed. It runs with NumPy 2.3.

import re
import numpy as np

# An invented COCO-style annotation: five captions and labelled boxes [x1, y1, x2, y2],
# normalized to [0, 1] as in LLaVA's prompts. Colors are extra labels for this example.
ANN = {
    "captions": ["a red bus waits at a stop next to a person",
                 "a person stands beside a red bus on a street",
                 "a city bus and a pedestrian near a bicycle",
                 "a bus stop with a red bus, a person and a parked bicycle",
                 "a person waits near a bus while a bicycle leans on a pole"],
    "objects": [("bus", "red", [0.05, 0.30, 0.60, 0.80]),
                ("person", "blue", [0.66, 0.42, 0.74, 0.85]),
                ("bicycle", "green", [0.78, 0.60, 0.97, 0.88])],
}

def render(ann, h=48, w=64):
    """Draw the annotation as a small RGB image (the pixels a VLM would see)."""
    rgb = {"red": (1, 0, 0), "blue": (0, 0, 1), "green": (0, 1, 0), "yellow": (1, 1, 0)}
    img = np.full((h, w, 3), 0.8)
    for _, color, (x1, y1, x2, y2) in ann["objects"]:
        img[int(y1 * h):int(y2 * h), int(x1 * w):int(x2 * w)] = rgb[color]
    return img

def context(ann):
    """The symbolic context sent to the text-only generator: captions + boxes."""
    lines = list(ann["captions"])
    lines += [f"{c} {n}: [{', '.join(f'{v:.2f}' for v in b)}]" for n, c, b in ann["objects"]]
    return "\n".join(lines)

def phrase(n, c):
    return f"{n} in {c}" if n == "person" else f"{c} {n}"

def side(b):
    x = (b[0] + b[2]) / 2
    return "left" if x < 1 / 3 else "middle" if x < 2 / 3 else "right"

def generate(ctx):
    """Stand-in for GPT-4: templates that read ONLY the context string, never pixels."""
    objs = [(m[2], m[1], [float(v) for v in m[3].split(", ")])
            for m in re.finditer(r"^(\w+) (\w+): \[(.*)\]$", ctx, re.M)]
    big = max(objs, key=lambda o: (o[2][2] - o[2][0]) * (o[2][3] - o[2][1]))
    conv = [(f"What color is the {n}?", f"The {n} is {c}.") if n != "person" else
            ("What color is the person wearing?", f"The person is wearing {c}.") for n, c, _ in objs]
    conv.append(("How many objects are labelled?", f"There are {len(objs)}."))
    detail = " ".join(f"A {phrase(n, c)} is on the {side(b)} side of the image." for n, c, b in objs)
    a, b = objs[0], objs[1]
    rel = "left of" if a[2][2] < b[2][0] else "right of" if a[2][0] > b[2][2] else "overlapping"
    reason = (f"The {phrase(big[0], big[1])} covers the largest area, so it is probably the main subject "
              f"or the closest object. The {a[0]} is {rel} the {b[0]}, which suggests the "
              f"{b[0]} is waiting beside the {a[0]}.")
    return {"conversation": conv,
            "detailed description": [("Describe the image in detail.", detail)],
            "complex reasoning": [("What is the person likely doing?", reason)]}

def tokens(text):
    """A rough tokenizer: words and punctuation marks."""
    return re.findall(r"\w+|[^\w\s]", text)

ctx = context(ANN)
samples = generate(ctx)
print(f"generator input: {len(tokens(ctx))} tokens of text, 0 pixels")
for kind, turns in samples.items():
    q, a = turns[-1] if kind != "conversation" else turns[0]
    print(f"\n[{kind}] {len(turns)} turn(s)\n  Q: {q}\n  A: {a}")

# Token budget of one training sequence per type, with the image as visual tokens.
print("\ntype                   visual  question  answer  supervised")
for n_vis, enc in [(256, "224 px"), (576, "336 px")]:
    for kind, turns in samples.items():
        nq = sum(len(tokens(q)) for q, _ in turns)
        na = sum(len(tokens(a)) for _, a in turns)
        print(f"{kind:21s} {n_vis:6d}  {nq:8d}  {na:6d}  {na / (n_vis + nq + na):9.1%}  ({enc})")

# The generator is blind. Suppose the photo also shows a dog that the annotators missed:
# the model is trained on these pixels, but no instruction text can mention the dog.
photo = render({"objects": ANN["objects"] + [("dog", "yellow", [0.40, 0.82, 0.50, 0.98])]})
text = " ".join(q + " " + a for turns in samples.values() for q, a in turns)
print(f"\ndog pixels in the training image: {(photo != render(ANN)).any(-1).sum()}, "
      f"mentions of 'dog' in its instruction data: {tokens(text).count('dog')}")

generator input: 117 tokens of text, 0 pixels

[conversation] 4 turn(s)
  Q: What color is the bus?
  A: The bus is red.

[detailed description] 1 turn(s)
  Q: Describe the image in detail.
  A: A red bus is on the left side of the image. A person in blue is on the right side of the image. A green bicycle is on the right side of the image.

[complex reasoning] 1 turn(s)
  Q: What is the person likely doing?
  A: The red bus covers the largest area, so it is probably the main subject or the closest object. The bus is left of the person, which suggests the person is waiting beside the bus.

type                   visual  question  answer  supervised
conversation             256        25      20       6.6%  (224 px)
detailed description     256         6      37      12.4%  (224 px)
complex reasoning        256         7      38      12.6%  (224 px)
conversation             576        25      20       3.2%  (336 px)
detailed description     576         6      37       6.0%  (336 px)
complex reasoning        576         7      38       6.1%  (336 px)

dog pixels in the training image: 56, mentions of 'dog' in its instruction data: 0

The generator saw 117 tokens of text and no pixels, so the dog, although in the training image, cannot appear in any answer, and a wrong label would pass straight into the answers. The reasoning answer also goes beyond its evidence: nothing in the boxes says the person is waiting. GPT-4 writes richer answers from the same text-only context. Answers fill only 3–13% of each sequence, and in LLaVA only answers (and stop markers) receive a loss.

Two-stage training

Each sample becomes a conversation: a system message, then turns of Human: instruction and Assistant: answer separated by a stop marker, with the image placed before or after the first question at random. With $X_{\text{instruct},<i}$ and $X_{a,<i}$ the instruction and answer tokens before position $i$, the model maximizes

$$p(X_a \mid X_v, X_{\text{instruct}}) = \prod_{i=1}^{L} p_\theta\big(x_i \mid X_v, X_{\text{instruct},<i}, X_{a,<i}\big),$$

where $\theta$ are the trainable parameters; only answer tokens and stop markers contribute to the loss (Liu et al., 2023).

1. Feature alignment. 595K image–caption pairs, filtered from Conceptual Captions (CC3M) for broad noun-phrase coverage, become single-turn captioning requests such as “Describe the image concisely”, answered with the caption. Only $\theta = W$ is trained; the authors call this training a compatible visual tokenizer for the frozen language model.
2. End-to-end fine-tuning. The encoder stays frozen and $\theta = \{W, \phi\}$, projector and language model, is trained on LLaVA-Instruct-158K for the chat model or on ScienceQA.

On eight A100 GPUs, stage 1 took under 4 hours (1 epoch) and stage 2 under 10 hours (3 epochs) (Liu et al., 2023, appendix). Skipping stage 1 cost 5.11 points on ScienceQA.

Example: does alignment pretraining help?

A tiny analogue on synthetic $32 \times 32$ images of four colored shapes. The frozen “vision encoder” is two random, never-trained convolution layers giving a $4 \times 4$ grid of 64-dimensional features; an invented 12-word vocabulary has frozen random embeddings and an output layer tied to them. Stage 1 trains only the projector $W$ on 2,000 captioned images, scoring the mean visual token against the caption words’ embeddings. Stage 2 trains $W$ and a one-layer causal decoder, randomly initialized rather than pretrained, on 300 images with templated questions. It uses PyTorch 2.14.1, runs in about 13 seconds over seeds 0–5, and prints identical output on repeated runs.

import torch
import torch.nn as nn
import torch.nn.functional as F

torch.set_num_threads(4)
S, D = 32, 32
COLORS = {"red": (1., .15, .15), "green": (.15, 1., .15), "blue": (.15, .15, 1.), "yellow": (1., 1., .15)}
SHAPES = ["circle", "square", "triangle", "bar"]
VOCAB = ["what", "color", "shape", "?"] + list(COLORS) + SHAPES        # invented vocabulary
IDX = {w: i for i, w in enumerate(VOCAB)}
C0, S0 = IDX["red"], IDX["circle"]                                     # first color / shape word
yy, xx = torch.meshgrid(torch.arange(S) + .5, torch.arange(S) + .5, indexing="ij")

def draw(color, shape):
    """A 32x32 RGB image with one colored shape at a random position and size."""
    cx, cy = torch.empty(2).uniform_(10, 22)
    r = float(torch.empty(1).uniform_(5, 8))
    x, y = xx - cx, yy - cy
    m = {"circle": x ** 2 + y ** 2 < r ** 2,
         "square": (x.abs() < .8 * r) & (y.abs() < .8 * r),
         "triangle": (y < .8 * r) & (y > -r + 2 * x.abs()),
         "bar": (x.abs() < 1.2 * r) & (y.abs() < .35 * r)}[shape]
    return .1 + m[None] * torch.tensor(COLORS[color])[:, None, None] + .05 * torch.randn(3, S, S)

def make_data(n):
    """Images with their two caption words ("red circle") as vocabulary ids."""
    c, s = torch.randint(4, (n,)), torch.randint(4, (n,))
    imgs = torch.stack([draw(list(COLORS)[i], SHAPES[j]) for i, j in zip(c, s)])
    return imgs, c + C0, s + S0

class TinyLLaVA(nn.Module):
    def __init__(self):
        super().__init__()
        # Frozen vision encoder: random convolutions, never trained -> 4x4 grid of 64-d features.
        self.encoder = nn.Sequential(nn.Conv2d(3, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
                                     nn.Conv2d(32, 64, 3, padding=1), nn.ReLU(), nn.MaxPool2d(4))
        self.embed = nn.Embedding(len(VOCAB), D)          # the "LLM" word embeddings, frozen
        self.encoder.requires_grad_(False)
        self.embed.requires_grad_(False)
        self.proj = nn.Linear(64, D)                       # connector W
        layer = nn.TransformerEncoderLayer(D, 4, 4 * D, dropout=0., batch_first=True, norm_first=True)
        self.decoder = nn.TransformerEncoder(layer, 1, enable_nested_tensor=False)
        self.pos = nn.Parameter(torch.randn(20, D) * .02)

    def features(self, img):                               # frozen, so computed once per image
        with torch.no_grad():
            return self.encoder(img).flatten(2).transpose(1, 2)          # (N, 16, 64)

    def logits(self, h):                                   # output head tied to the frozen embeddings
        return h @ self.embed.weight.T

    def match(self, feats):
        """Stage 1: score words against the mean projected visual token."""
        return self.logits(self.proj(feats).mean(1))

    def answer(self, feats, question):
        """Stage 2: visual tokens, then the question, through a causal decoder."""
        x = torch.cat([self.proj(feats), self.embed(question)], 1) + self.pos[: 16 + question.shape[1]]
        mask = nn.Transformer.generate_square_subsequent_mask(x.shape[1])
        return self.logits(self.decoder(x, mask=mask, is_causal=True)[:, -1])

def ask(n, kind):
    return torch.tensor([IDX["what"], IDX[kind], IDX["?"]]).expand(n, 3)   # "what color ?"

def accuracy(model, te):
    with torch.no_grad():
        c = (model.answer(te[0], ask(len(te[1]), "color")).argmax(1) == te[1]).float().mean()
        s = (model.answer(te[0], ask(len(te[1]), "shape")).argmax(1) == te[2]).float().mean()
    return c.item(), s.item()

CHECK = [25, 50, 100, 200]

def run(seed, use_stage1, n_qa=300):
    torch.manual_seed(seed)
    model = TinyLLaVA()
    cap, qa, te = make_data(2000), make_data(n_qa), make_data(1000)   # many captions, few Q&A
    cap, qa, te = [(model.features(d[0]),) + d[1:] for d in (cap, qa, te)]
    matched = None
    if use_stage1:                                         # stage 1: train W only
        opt = torch.optim.Adam(model.proj.parameters(), lr=1e-2)
        for step in range(200):
            i = torch.randint(2000, (128,))
            out = model.match(cap[0][i])
            loss = F.cross_entropy(out, cap[1][i]) + F.cross_entropy(out, cap[2][i])
            opt.zero_grad(); loss.backward(); opt.step()
        with torch.no_grad():                              # zero-shot caption matching on test images
            sc = model.match(te[0])
            matched = ((sc[:, C0:C0 + 4].argmax(1) + C0 == te[1]).float().mean().item(),
                       (sc[:, S0:S0 + 4].argmax(1) + S0 == te[2]).float().mean().item())
    # stage 2: W + decoder on templated Q&A
    opt = torch.optim.Adam([p for p in model.parameters() if p.requires_grad], lr=3e-3)
    curve = []
    for step in range(1, CHECK[-1] + 1):
        i = torch.randint(n_qa, (64,))
        is_color = torch.rand(64) < .5
        q = torch.where(is_color[:, None], ask(64, "color"), ask(64, "shape"))
        y = torch.where(is_color, qa[1][i], qa[2][i])
        loss = F.cross_entropy(model.answer(qa[0][i], q), y)
        opt.zero_grad(); loss.backward(); opt.step()
        if step in CHECK:
            curve.append(accuracy(model, te))
    return torch.tensor(curve), matched                    # (checkpoints, [color, shape])

print(f"{'stage-2 steps':29s}" + "".join(f"{s:>18d}" for s in CHECK))
for use_stage1, name in [(False, "stage 2 only"), (True, "stage 1 + stage 2")]:
    runs = [run(seed, use_stage1) for seed in range(6)]
    acc = torch.stack([r[0] for r in runs]).mean(-1)       # mean of color and shape, (seeds, checks)
    cells = "".join(f"  {m:.2f} [{lo:.2f}-{hi:.2f}]" for m, lo, hi in
                    zip(acc.mean(0), acc.min(0).values, acc.max(0).values))
    print(f"{name:18s} accuracy  {cells}")
    final = torch.stack([r[0][-1] for r in runs]).mean(0)
    print(f"{'':18s} at 200 steps: color {final[0]:.2f}, shape {final[1]:.2f}")
    if use_stage1:
        m = torch.tensor([r[1] for r in runs]).mean(0)
        print(f"{'':18s} after stage 1 alone, caption matching: color {m[0]:.2f}, shape {m[1]:.2f}")

stage-2 steps                                25                50               100               200
stage 2 only       accuracy    0.21 [0.13-0.30]  0.38 [0.25-0.49]  0.61 [0.38-0.69]  0.79 [0.69-0.84]
                   at 200 steps: color 1.00, shape 0.59
stage 1 + stage 2  accuracy    0.41 [0.31-0.56]  0.60 [0.45-0.68]  0.72 [0.64-0.81]  0.82 [0.76-0.87]
                   at 200 steps: color 1.00, shape 0.65
                   after stage 1 alone, caption matching: color 1.00, shape 0.58

After 25 stage-2 steps the aligned model answers 41% correctly against 21% without alignment (chance is 25%; an untrained model also outputs non-answer words). By 200 steps the gap shrinks to 0.82 against 0.79, with overlapping ranges over seeds. Stage 1 alone already matches colors perfectly and shapes 58% of the time. Here alignment mainly speeds up stage 2; shape stays hard, probably because frozen random features pooled to a $4 \times 4$ grid keep little of an outline.

LLaVA-1.5: academic data and format prompts

The original model was weak on visual question answering benchmarks that expect one-word answers and tended to answer “yes” to yes/no questions; InstructBLIP answered briefly even when asked for detail. LLaVA-1.5 appends a response format prompt to short-answer questions, such as “Answer the question using a single word or phrase.” (Liu et al., 2024), so the model learns when to be brief. Its 665K-sample instruction mix adds VQAv2, GQA, OKVQA, A-OKVQA, OCRVQA, TextCaps, RefCOCO, Visual Genome and 40K text-only ShareGPT conversations to LLaVA-Instruct-158K (arXiv version, Table 7); stage 1 uses 558K caption pairs. In the 7B ablation at 224 pixels, adding VQAv2 raised MME perception from 809.6 to 1197.0 and the format prompt to 1323.8; the full 13B recipe reached 1531.3. Training takes about 6 plus 20 hours on eight A100s, using only public data.

Inference

The input is one image and a text instruction; the output is free text, decoded greedily in LLaVA-1.5’s evaluations (arXiv appendix). A $336 \times 336$ image adds 576 tokens to every forward pass. For benchmarks, LLaVA-1.5 appends the format prompt matching the answer type; a prompt allowing “Unanswerable” raised the score on VizWiz’s unanswerable questions from 11.1% to 67.8% (Liu et al., 2024).

Performance

LLaVA-Bench. Text-only GPT-4 rates LLaVA’s answer and a reference answer written by GPT-4 from ground-truth text (captions and boxes for COCO, curated descriptions otherwise) from 1 to 10 for helpfulness, relevance, accuracy and detail; LLaVA’s score is reported relative to the reference. On LLaVA-Bench (COCO), 30 val2014 images with 90 questions, LLaVA scored 85.1%, and 21.5% without instruction tuning. On LLaVA-Bench (In-the-Wild), 24 images including memes, paintings and sketches with 60 questions, it scored 67.3% against 38.1% for BLIP-2 and 19.1% for OpenFlamingo (Liu et al., 2023).

ScienceQA. Fine-tuned on this 21K-question multiple-choice science benchmark, LLaVA reached 90.92%, close to the previous best of 91.68% (MM-CoT Large); text-only GPT-4 with two examples reached 82.69%. Asking GPT-4 to settle disagreements between the two gave 92.53%, a new state of the art at the time (Liu et al., 2023).

LLaVA-1.5. LLaVA-1.5-13B reported, among others, 80.0 on VQAv2 and 63.3 on GQA (training images seen), 61.3 on TextVQA, 71.6 on ScienceQA-IMG, POPE F1 of 87.1, 86.2 and 84.5 on the random, popular and adversarial splits, 67.7 on MMBench and 36.1 on MM-Vet (Liu et al., 2024, Tables 3–4). The paper evaluates 12 benchmarks; its abstract claims state of the art on 11.

Strengths

• Simplicity. A matrix or two-layer MLP between two pretrained models; LLaVA-1.5’s authors argue their results question the need for resamplers and large-scale alignment pretraining.
• Efficiency. LLaVA-1.5 used 1.2M public samples and one day on one 8-GPU node, against 129M pretraining pairs for InstructBLIP and 1.4B for Qwen-VL; half of its instruction mix kept over 98% of full performance (Liu et al., 2024).
• Openness. Data, code, checkpoints and a demo were released.

Limitations

• Hallucination. LLaVA tends to say “yes”: its POPE F1 (7B) was 76.3, 72.2 and 70.1, against 87.3, 86.1 and 84.2 for LLaVA-1.5-7B (Liu et al., 2024). It confirmed strawberry-flavored yogurt in a fridge holding yogurt and strawberries, which its authors read as perceiving the image as an unordered set of patches (Liu et al., 2023). Multimodal Hallucination covers how it is measured and reduced.
• Generated data. The data inherits the generator’s blind spots, as in the example; LLaVA-1.5’s authors note some hallucinated content in the detailed descriptions, observe less hallucination at $448$-pixel input, and argue that models learn to hallucinate when training text describes detail their resolution cannot resolve.
• Judge-based evaluation. LLaVA-Bench depends on a proprietary judge that sees text instead of the image, and its sets are small.
• Resolution and scope. At 224 or 336 pixels small text is lost (reading a yogurt brand is among LLaVA-Bench’s hard cases); LLaVA-1.5 handles one image at a time, and its authors warn against critical uses such as medicine.

Historical Importance

LLaVA showed that machine-generated instruction data and a linear projector could make an open model follow visual instructions; LLaVA-1.5 showed that the same design with better data matched or beat models with resamplers and far more pretraining. By LLaVA-1.5’s account, the architecture had already been reused for region- and pixel-level understanding, biomedical assistants and image generation, and the data pipeline extended to text-rich images and region-level dialogue (Liu et al., 2024).

Variants

Version	Year	Vision encoder	Connector	Language model	Training data	Key results
LLaVA	2023	CLIP ViT-L/14, frozen (224 px per LLaVA-1.5)	linear	Vicuna (13B; 7B ablation)	CC3M 595K; LLaVA-Instruct-158K	LLaVA-Bench COCO 85.1%, In-the-Wild 67.3% (relative); ScienceQA 90.92% fine-tuned
LLaVA-Med	2023	as LLaVA, initialized from it	linear	as LLaVA	PMC-15M 600K; 60K GPT-4 instructions	trained in under 15 h on eight A100s; fine-tuned on VQA-RAD, SLAKE, PathVQA
LLaVA-1.5	2024	CLIP ViT-L/14, 336 px, frozen	two-layer MLP	Vicuna-v1.5 7B, 13B	558K; 665K mix	13B: VQAv2 80.0, GQA 63.3, TextVQA 61.3, MM-Vet 36.1
LLaVA-1.5-HD	2024	CLIP ViT-L/14, 224 px tiles (up to 6) + global view	two-layer MLP	Vicuna-v1.5 13B	as LLaVA-1.5	VQAv2 81.8, TextVQA 62.5, MM-Vet 39.4
LLaVA-NeXT	2024	AnyRes tiling	—	best open LLM of the time	expanded instruction data	released as blog posts
LLaVA-OneVision	2025	SigLIP, 384 px, AnyRes, trained	two-layer MLP	Qwen-2 0.5B, 7B, 72B	558K; 4M; 3.2M; 1.6M	72B: MMMU val 56.8, DocVQA test 91.3

Years are those of the cited publications; LLaVA-1.5 and LLaVA-OneVision first appeared on arXiv in 2023 and 2024. Numbers are as printed; for VQAv2 and GQA, training images were seen.

LLaVA-1.5-HD tiles the image into up to six $224 \times 224$ crops and adds a downsampled global view, as described in the Vision–Language Model article; it needs no extra pretraining (Liu et al., 2024, arXiv appendix). Its text says HD improves overall performance on all benchmarks, but Tables 3–4 show some scores falling, MME from 1531.3 to 1500.1 and ScienceQA-IMG from 71.6 to 71.0.

LLaVA-NeXT appeared as blog posts in 2024; the OneVision paper summarizes its changes as AnyRes tiling for high-resolution images, more high-quality instruction data and the best open LLM available, with later posts adding video, larger language models and interleaved images.

LLaVA-OneVision handles single images, multiple images and video in one model. Each $384 \times 384$ view gives 729 SigLIP tokens, and the maximum token budgets of the three scenarios are kept similar (6,272 to 8,748 in the paper’s examples). A stage 1.5 of 4M high-quality “knowledge” samples sits between alignment and instruction tuning, which uses 3.2M single-image samples and then 1.6M mixed single-image, multi-image and video samples. After stage 1 the vision encoder is also trained, at a fifth of the LLM’s learning rate (Li et al., 2025, arXiv version).

LLaVA-Med adapts LLaVA to biomedicine, as a domain-adaptation recipe: starting from LLaVA, stage 1 aligns the projector on 600K figure–caption pairs from PubMed Central (PMC-15M), and stage 2 tunes on 60K GPT-4-generated instructions; the stages took 7 and 8 hours on eight 40GB A100s (Li et al., 2023).

Implementations

The authors release code, data and checkpoints through the project page (llava-vl.github.io) and its GitHub repository; LLaVA-OneVision’s evaluations use the LMMs-Eval toolkit.