Instruction Tuning for Vision–Language Models

Instruction tuning for vision–language models, usually called visual instruction tuning, is the stage that turns a model which can describe images into one that does what a user asks about an image. A pretrained vision–language model is fine-tuned on (image, instruction, response) examples with the language-modelling loss on the responses, and is often then tuned on pairs of preferred and rejected responses. The term is the title of the 2023 paper that introduced LLaVA (Visual Instruction Tuning), whose authors called it the first attempt to extend instruction tuning to the language–image space; a text-only GPT-4 wrote its training data from COCO captions and boxes. Its two families of methods are covered here: building and mixing instruction data for supervised fine-tuning, and preference tuning with human or AI feedback (LLaVA-RLHF, RLHF-V, RLAIF-V). The model articles keep their architectures: LLaVA for the projector design, BLIP for InstructBLIP’s Q-Former, and Vision–Language Pretraining for the stage before.

Purpose

Alignment pretraining on image–caption pairs teaches a model to map image features into the language model’s input space, but every training target is a caption. Asked a question, such a model tends to describe the image instead. The LLaVA article gives the original example: BLIP-2 and OpenFlamingo described a photo of a man ironing on a moving car when asked what was unusual about it.

Instruction tuning addresses three things. It teaches the model to read the instruction and to answer in the requested form (one word, a sentence, a paragraph, a letter from a list of options). It widens the set of tasks the model has seen and states each as an instruction; InstructBLIP’s authors found the instruction format, rather than multitask training on the same datasets, to be what carries over to datasets and task types left out of training (Dai et al., 2023). And, through preference tuning, it shapes which of the plausible answers the model prefers, most often the one that claims less that the image does not show (Sun et al., 2024; Yu et al., 2024).

When to Use It

Instruction tuning is the stage to use when a model must take open requests about images: chat assistants, visual question answering with a natural-language interface, description at a chosen level of detail. It assumes a model that already connects images to its language model, usually after an alignment stage.

It helps less, or needs care, in three situations:

• One fixed task. A model that only ever answers one benchmark question type can be fine-tuned on that task directly. Instruction formats matter when the request varies.
• A small or frozen language model. If only a small bridge module is trained, it has to control how long the language model’s answer is. LLaVA-1.5’s authors attribute InstructBLIP’s tendency to answer long requests with one word partly to this (Liu et al., 2024).
• Hallucination as the main problem. Supervised data can itself teach a model to state things it cannot see (see Pitfalls). Preference tuning, data cleaning or higher input resolution are then the relevant levers, and Multimodal Hallucination surveys them.

How It Works

Generating instruction data

Three ways of producing the (image, instruction, response) triples have been used, and they differ in who writes the response and whether that writer sees the image.

A text-only language model writing from annotations. LLaVA gave GPT-4 the five captions and the object boxes of a COCO image, with a few hand-written seed examples, and asked for conversations, detailed descriptions and complex-reasoning answers (Liu et al., 2023). The LLaVA paper article describes the prompts and which of them include boxes, and the LLaVA article runs a small version of the pipeline. The official repository releases the prompts and few-shot examples along with the 58K conversations, 23K descriptions and 77K reasoning samples (LLaVA repository, docs/Data.md). Because the writer is blind, anything the annotations omit cannot appear in a response, and anything the writer infers beyond them can enter the response as fact.

Templates over existing datasets. InstructBLIP converted 26 public datasets in 11 task categories, written for captioning, VQA, OCR, video QA and classification, into instruction format, with 10 to 15 hand-written templates per task. For datasets whose answers are short, some templates contain words such as “short” or “briefly”, so that the model does not learn to answer briefly whatever it is asked. LLaVA-Instruct-150K was one of the 13 training datasets and kept its own format (Dai et al., 2023). In our reading, the human-written responses of these datasets are reliable but short. LLaVA-RLHF’s authors used the same route to add yes/no questions from VQA-v2, multiple-choice questions from A-OKVQA and grounded captions from Flickr30k to LLaVA’s data (Sun et al., 2024).

A stronger vision–language model as annotator. ShareGPT4V had GPT-4V, which sees the image, write 100K detailed captions, with a base prompt plus source-specific prompts for artworks, landmarks, celebrities and text-heavy images. The images came from COCO (50K), LAION, CC and SBU (30K), SAM (20K) and smaller web sources (supplementary material). The captions average 942 characters, against 609 for LLaVA’s 23K GPT-4 descriptions. The authors then fine-tuned an open model on these captions as Share-Captioner and used it to caption 1.2M images (1,246K in their Table 2) in about 44 A100 GPU-days, which removes the cost of the proprietary annotator at scale (Chen et al., 2024). The released captions are licensed CC BY-NC 4.0 for research use only, as of November 2026 (ShareGPT4V repository README).

The choice matters measurably. ShareGPT4V’s authors replaced the 23K detailed descriptions in LLaVA-1.5-7B’s instruction data with 23K samples from each source in turn (their Table 1):

Replacement for LLaVA-1.5’s 23K descriptions	MME	MMBench	MM-Vet	LLaVA-Bench (In-the-Wild)
GPT-4 captions (LLaVA’s own, the baseline)	1510.7	64.3	30.5	63.4
BLIP-2 captions	1453.9	63.7	28.8	61.5
GPT-4V question–answer pairs	1494.4	65.0	31.8	67.7
GPT-4V detailed captions	1516.9	65.3	34.0	71.6

Short captions did worse than the original data, and GPT-4V descriptions did best on most columns. The text says the detailed captions beat the GPT-4V question–answer data on all VQA benchmarks, but on SEED (not in the table above) the table gives 66.9 for the question–answer data and 66.8 for the captions. To separate detail from accuracy, the supplementary material truncates the GPT-4V captions to LLaVA-1.5’s average caption length: MM-Vet rose from 30.5 to 32.3 with truncated captions and to 34.0 with full ones, which the authors read as both fewer hallucinations and more detail helping, detail slightly more (Chen et al., 2024).

Supervised fine-tuning: what is trained and what is frozen

The loss is the one used in pretraining, restricted to the response: the model maximizes the probability of the answer tokens given the image tokens, the instruction and the earlier answer tokens. The LLaVA article writes it out; instruction and padding tokens get no loss. What changes from model to model is which parameters move:

Model	Vision encoder	Connector	Language model	Notes
LLaVA (2023)	frozen	trained	fully trained	stage 2 after projector-only alignment
InstructBLIP (2023)	frozen	Q-Former trained	frozen	initialized from BLIP-2; up to 60K steps
LLaVA-1.5 (2024)	frozen in the official code (not stated in the paper)	two-layer MLP trained	fully trained	2e-5 learning rate, one epoch, batch 128
ShareGPT4V-7B (2024)	latter half trained in pretraining, frozen in SFT	trained	trained in both stages	pretraining at 2e-5 on 1.2M captions
LLaVA-RLHF (2024)	not stated	initialized from LLaVA’s linear projector	fine-tuned in SFT; LoRA (rank 64) in the RL stage	reward and value models also LoRA-tuned

Sources: Liu et al. (2023); Dai et al. (2023); Liu et al. (2024, supplementary Table 3); Chen et al. (2024); Sun et al. (2024, appendix I); for LLaVA-1.5’s encoder, the LLaVA repository’s CLIP encoder module, which disables its gradients, and its v1.5 fine-tuning script. LLaVA-1.5’s supplementary text says its hyperparameters are LLaVA’s except a halved pretraining learning rate, but its Table 3 lists one epoch of instruction tuning and batch sizes of 256 and 128, where the original LLaVA trained for three epochs with batch sizes of 128 and 32.

Two findings bear on the choice. LLaVA-1.5’s authors argue that freezing the language model makes format control harder, because a small module must then steer whether the frozen model answers briefly or at length (Liu et al., 2024). ShareGPT4V’s authors argue the opposite case for the encoder: freezing it was reasonable for earlier models given the lower quality of their captions, but with detailed captions, unfreezing the second half of the CLIP encoder during pretraining gave the best of five settings, with MME perception 1567.4 against 1515.2 with the encoder frozen and 1545.7 with all 24 blocks trained, at 56.7 GB against 49.6 GB and 63.6 GB of memory (their Table 8). Fine-Tuning covers the general trade-offs of freezing, discriminative learning rates and forgetting.

Data mixtures and the format problem

Mixing academic datasets with conversational data raises a problem that does not exist for a single dataset: the same kind of question must sometimes be answered with one word and sometimes with a paragraph. LLaVA-1.5’s authors give two main causes of InstructBLIP’s short answers to long requests: ambiguous prompts and, as above, the frozen language model. With a template such as Q: {Question} A: {Answer}, nothing tells the model which length is wanted, and VQA data teaches it to be brief everywhere (Liu et al., 2024). Their fix is a response format prompt: one fixed sentence appended to every short-answer question, “Answer the question using a single word or phrase.” Other datasets get their own sentence: answer with the option’s letter for A-OKVQA’s multiple choice, give a one-sentence caption for TextCaps (supplementary Table 1). With the format stated, the model answers briefly when asked to and at length otherwise, and the authors show it following format requests it was not trained on.

LLaVA-1.5’s ablation (Table 2, 7B at 224 pixels) shows both sides. Adding VQAv2 raised MME from 809.6 to 1197.0, and adding the format prompt raised it to 1323.8, but the format prompt also lowered GQA from 47.0 to 46.8 and MM-Vet from 27.7 to 26.3. The text credits 1323.8 against 809.6 to including VQAv2 alone; the table assigns 1197.0 to VQAv2 and 1323.8 to VQAv2 plus the format prompt. The final 665K mixture, from 158K LLaVA conversations, 40K text-only ShareGPT conversations and eight academic datasets, is listed in the LLaVA article. The supplementary material adds how it was assembled: all question–answer pairs about one image are merged into one conversation; each A-OKVQA question is repeated once per answer option, to counter the lack of multiple-choice data; each batch draws from only one modality (image or text-only), which sped training by 25% without changing the result; and the parts are concatenated and sampled uniformly, so each dataset contributes in proportion to its size (Liu et al., 2024).

InstructBLIP balanced differently. Sampling uniformly over many datasets of very different sizes made the model overfit the small ones and underfit the large ones, so the authors sampled dataset $d$ with probability

$$p_d = \frac{\sqrt{S_d}}{\sum_{i=1}^{D} \sqrt{S_i}},$$

where $S_d$ is its number of training samples, and then adjusted some weights by hand (lowering A-OKVQA, raising OKVQA) according to how fast each converged (Dai et al., 2023). Without balancing, held-out ScienceQA (image context) fell from 70.4 to 66.0 for the FlanT5-XL model; for Vicuna-7B the drop was smaller, and IconQA rose slightly, from 43.1 to 43.5 (Table 2).

Two further results show that the right mixture is not simply “more datasets”. LLaVA-1.5 kept more than 98% of its full performance when its mixture was randomly subsampled to half, and its MMBench, ScienceQA and POPE scores did not drop at 50% (Liu et al., 2024). And in LLaVA-RLHF’s ablation (Table 5, 7B), adding any one of the three datasets to LLaVA’s conversations lowered the LLaVA-Bench score sharply, while VQA-v2 raised POPE (yes/no) most and A-OKVQA raised MMBench (multiple choice) most; only all three together raised both kinds of score:

Added to LLaVA’s conversations	MMBench	POPE F1	LLaVA-Bench	MMHal-Bench
none (LLaVA)	38.7	76.0	81.0	1.3
VQA-v2 yes/no	42.9	82.0	30.4	2.0
A-OKVQA multiple choice	48.5	79.8	34.7	1.1
Flickr30k grounded captions	37.8	77.6	46.6	1.5
all three (LLaVA-SFT+)	52.1	82.7	86.3	1.8

The appendix text says Flickr30k notably improves LLaVA-Bench and MMHal-Bench; the table shows MMHal-Bench rising but LLaVA-Bench falling from 81.0 to 46.6 (Sun et al., 2024). Instruction data also transfers across modalities: LLaVA-1.5 follows instructions in other languages although all its visual instructions are English, which its authors attribute partly to the multilingual text-only ShareGPT conversations in the mixture (Liu et al., 2024).

Preference tuning

Supervised fine-tuning imitates reference responses, so it cannot express that one plausible answer is worse than another. Preference tuning trains on comparisons. Given an image and prompt $x$, a preferred response $y_w$ and a rejected one $y_l$, the model is pushed towards $y_w$ while being kept close to the supervised model $\pi_{\text{ref}}$.

RLHF with a reward model. LLaVA-RLHF, described by its authors as the first large multimodal model trained with RLHF, follows the text recipe in three stages (Sun et al., 2024). Crowdworkers compared two responses sampled at temperature 0.7 for 10K held-out LLaVA conversations, choosing first the one with fewer hallucinations and, only if tied, the more helpful one. A reward model $r_\theta$, initialized from the supervised model with a scalar head, was trained for one epoch on

$$\mathcal{L}(r_\theta) = -\mathbb{E}\big[\log \sigma\big(r_\theta(I, x, y_w) - r_\theta(I, x, y_l)\big)\big],$$

reaching 65% validation accuracy, against 69% agreement between human labelers. The policy was then optimized with PPO to maximize $r_\theta$ minus a per-token KL penalty to the initial policy (coefficient 0.1), on 50K held-out conversations (first question only, to avoid hallucinations already in the conversation context), 12K A-OKVQA questions and 10K VQA-v2 questions.

The reward model was the weak link. With ordinary RLHF, the policy learned to write longer responses, which the reward model scored higher, without hallucinating less. Factually Augmented RLHF (Fact-RLHF) gives the reward model the image’s five COCO captions, or three A-OKVQA rationales, in its prompt, during both training and scoring, so that it can catch claims the captions contradict. It adds symbolic rewards: a correctness penalty where the answer is known (yes/no, multiple choice), set in the appendix to 2 for correct and 0 for incorrect answers, and a length penalty with coefficients set per question type (appendix I). In the 7B ablation with a 13B reward model, Fact-RLHF raised MMHal-Bench from 1.8 to 2.1 and LLaVA-Bench from 93.4 to 94.1 over plain RLHF. Against the supervised LLaVA-SFT+ model, RLHF at 7B lowered MMBench slightly (52.1 to 51.4), which the authors compare to the “alignment tax” seen in language models, while the 13B model gained (57.5 to 60.1). Filtering LLaVA’s data with the reward model instead of running RL left MMHal-Bench, POPE and MMBench largely unchanged (Sun et al., 2024).

Direct preference optimization. DPO removes the reward model and the RL loop. Under the KL-constrained objective, the optimal policy defines an implicit reward $\hat{r}(x, y) = \beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)}$, and substituting it into the Bradley–Terry preference model gives a loss on the policy alone (Rafailov et al., 2023):

$$\mathcal{L}_{\text{DPO}} = -\mathbb{E}_{(x, y_w, y_l)}\left[\log \sigma\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\right)\right].$$

Its gradient raises the likelihood of $y_w$ and lowers that of $y_l$, weighted by how wrongly the implicit reward orders the pair.

Dense DPO on human corrections. RLHF-V changes both the feedback and the loss (Yu et al., 2024). Instead of ranking two long responses, annotators corrected the hallucinated segments of one response, so $y_w$ is the corrected text and $y_l$ the original, and they differ only where the correction is. The authors argue that ranking long responses is ambiguous, and that whole-response labels mix the preferred behaviour with what they call linguistic variance and non-robust bias, which the model can learn instead. 1.4K responses from their base model Muffin were annotated, averaging 64.4 words and 2.65 corrected segments. DDPO (dense DPO) replaces the sequence log-likelihood in the DPO loss with a weighted average that up-weights the corrected segments $y_c$ against the unchanged ones $y_u$:

$$\log \pi(y \mid x) = \frac{1}{N}\Big(\sum_{y_i \in y_u} \log p(y_i \mid x, y_{<i}) + \gamma \sum_{y_i \in y_c} \log p(y_i \mid x, y_{<i})\Big), \qquad N = |y_u| + \gamma |y_c|,$$

with $\gamma = 5$ and $\beta = 0.5$; training took under an hour on eight A100s. The normalization by $N$ keeps longer responses from scoring higher. In the ablation (Table 3), plain DPO gave 57.5% of MHumanEval responses with hallucinations against 55.5% for DDPO and 74.7% for the base Muffin, and more position errors, though fewer number errors (11.6% against 14.4%); the text sums this up as plain DPO degrading performance, which holds against DDPO on most columns but not against the base model. The authors also report that 200 of their correction pairs matched the hallucination rate reached with an order of magnitude more of LLaVA-RLHF’s ranking pairs (Figure 2).

AI feedback from open models. RLAIF-V replaces the human annotator with an open-source model rather than a stronger proprietary one, in the extreme case the model being trained (Yu et al., 2025). Two steps make a weak labeler usable. Deconfounded generation samples all candidate responses from the same model with the same decoding settings and different seeds, so that a pair differs in content rather than style. Divide and conquer has a language model split each response into atomic claims, turns each claim into a yes/no question (“Does the clock read around 11:20?”), and asks the labeler model for the probabilities of “yes” and “no”. A response’s score is minus the number of claims for which “no” is more likely. Pairs with different scores are kept (at most two per instruction), with too-short preferred responses dropped so that the mean length difference stays under one word. Training is iterative DPO: each of four rounds samples new responses from the current model for 4K instructions, labels them and trains on them. With LLaVA-NeXT 34B as the labeler for LLaVA-1.5 7B, the share of Object HalBench responses with a hallucinated object fell from 54.5% to 10.5%, the 80.7% relative reduction quoted in the abstract, and MHumanEval’s response-level rate from 67.1% to 44.5%, while MMStar rose from 33.3 to 35.4. The ablations are telling: replacing divide-and-conquer with an overall quality score from the same labeler dropped the human agreement of the constructed pairs from 96.7% to 66.7%, and replacing the sampled preferred responses with human-written ones made results worse, which the authors hypothesize adds non-robust shallow patterns to the training data (Tables 2–3). The released preference set holds 83,132 pairs, licensed CC BY-NC 4.0, as of November 2026 (RLAIF-V repository README). A model aligned this way also serves as its own length-normalized reward, $\frac{\beta}{T} \log \frac{\pi_\theta(y)}{\pi_{\text{ref}}(y)}$ for a $T$-token response, which the authors use for best-of-$N$ selection at inference.

How to Apply It

Example: caption-only, instruction and mixed supervision

A tiny model is trained from scratch three times on the same synthetic images, differing only in the format of its supervision: captions only, instructions only, or half and half. Each $32 \times 32$ image shows one shape with three attributes: color (red, green, blue, yellow), shape (circle, square, triangle, cross) and side (left, right). An instruction lists the attributes it wants, in order, and the answer gives their values in that order: shape color ? → circle red <eos>. The caption instruction is the single token describe, answered red circle left <eos>. Six instruction types are used for training; three reordered pairs and two triples are held out, and describe is held out from the instruction-only model. The model has an $8 \times 8$ patch embedding giving 16 visual tokens, a two-layer causal transformer and a loss on answer tokens only. All three attributes are known by construction, so every answer can be scored exactly.

Answers are scored two ways, because the choice changes what looks good. Strict requires the exact answer, order and end token included. Lenient only requires every requested value to appear somewhere, the kind of check that rewards a model that ignores the instruction and lists everything. The code uses PyTorch 2.14.1 with four threads, runs seeds 0–2 for each condition, takes about two and a half minutes, and printed identical output on two runs.

import torch
import torch.nn as nn
import torch.nn.functional as F

torch.set_num_threads(4)
S, D = 32, 64
COLORS = {"red": (1., .15, .15), "green": (.15, 1., .15), "blue": (.15, .15, 1.), "yellow": (1., 1., .15)}
SHAPES = ["circle", "square", "triangle", "cross"]
SIDES = ["left", "right"]
ATTR = {"color": list(COLORS), "shape": SHAPES, "side": SIDES}
VOCAB = ["<pad>", "<eos>", "?", "describe"] + list(ATTR) + list(COLORS) + SHAPES + SIDES
IDX = {w: i for i, w in enumerate(VOCAB)}
HELD_IN = [("color",), ("shape",), ("side",), ("color", "shape"), ("shape", "color"), ("color", "side")]
HELD_OUT = [("side", "color"), ("shape", "side"), ("side", "shape"),
            ("side", "shape", "color"), ("shape", "side", "color")]
CAPTION = ("color", "shape", "side")           # what "describe" asks for, in caption order
yy, xx = torch.meshgrid(torch.arange(S) + .5, torch.arange(S) + .5, indexing="ij")

def draw(c, s, side):
    """One colored shape in the left or right half of a 32x32 image."""
    cx = float(torch.empty(1).uniform_(7, 11)) + 16 * side
    cy, r = float(torch.empty(1).uniform_(8, 24)), float(torch.empty(1).uniform_(4, 6.5))
    x, y = xx - cx, yy - cy
    m = {"circle": x ** 2 + y ** 2 < r ** 2,
         "square": (x.abs() < .8 * r) & (y.abs() < .8 * r),
         "triangle": (y < .8 * r) & (y > -r + 2 * x.abs()),
         "cross": ((x.abs() < .3 * r) & (y.abs() < r)) | ((y.abs() < .3 * r) & (x.abs() < r))}[SHAPES[s]]
    return .1 + m[None] * torch.tensor(COLORS[list(COLORS)[c]])[:, None, None] + .05 * torch.randn(3, S, S)

def make_images(n):
    lab = torch.stack([torch.randint(4, (n,)), torch.randint(4, (n,)), torch.randint(2, (n,))], 1)
    return torch.stack([draw(*map(int, l)) for l in lab]), lab

def encode(lab, kind):
    """kind is "describe" or a tuple of attribute names -> (instruction ids, answer ids)."""
    words = ["describe"] if kind == "describe" else list(kind) + ["?"]
    asked = CAPTION if kind == "describe" else kind
    ans = [ATTR[a][int(lab[list(ATTR).index(a)])] for a in asked] + ["<eos>"]
    pad = lambda w, n, left: (["<pad>"] * (n - len(w)) + w) if left else (w + ["<pad>"] * (n - len(w)))
    return [IDX[w] for w in pad(words, 4, True)], [IDX[w] for w in pad(ans, 4, False)]

class TinyVLM(nn.Module):
    def __init__(self):
        super().__init__()
        self.patch = nn.Conv2d(3, D, 8, stride=8)             # 16 visual tokens, trained from scratch
        self.tok = nn.Embedding(len(VOCAB), D)
        self.pos = nn.Parameter(torch.randn(24, D) * .02)
        layer = nn.TransformerEncoderLayer(D, 4, 4 * D, dropout=0., batch_first=True, norm_first=True)
        self.blocks = nn.TransformerEncoder(layer, 2, enable_nested_tensor=False)
        self.head = nn.Linear(D, len(VOCAB))

    def forward(self, img, text):                             # text: instruction (4) + answer so far
        v = self.patch(img).flatten(2).transpose(1, 2)
        x = torch.cat([v, self.tok(text)], 1)
        x = x + self.pos[: x.shape[1]]
        mask = nn.Transformer.generate_square_subsequent_mask(x.shape[1])
        return self.head(self.blocks(x, mask=mask, is_causal=True))

def batch(imgs, labs, idx, p_caption):
    kinds = ["describe" if torch.rand(1) < p_caption else HELD_IN[int(torch.randint(len(HELD_IN), (1,)))]
             for _ in idx]
    ins, ans = zip(*[encode(labs[i], k) for i, k in zip(idx.tolist(), kinds)])
    return imgs[idx], torch.tensor(ins), torch.tensor(ans)

def train(imgs, labs, p_caption, steps, bs=64):
    model = TinyVLM()
    opt = torch.optim.AdamW(model.parameters(), lr=2e-3, weight_decay=0.01)
    for step in range(steps):
        im, ins, ans = batch(imgs, labs, torch.randint(len(imgs), (bs,)), p_caption)
        text = torch.cat([ins, ans[:, :-1]], 1)                       # teacher forcing
        logits = model(im, text)[:, 16 + 3:]                          # positions that predict answers
        loss = F.cross_entropy(logits.reshape(-1, len(VOCAB)), ans.reshape(-1), ignore_index=IDX["<pad>"])
        opt.zero_grad(); loss.backward(); opt.step()
    return model

@torch.no_grad()
def generate(model, img, ins):
    text = ins
    for _ in range(4):                                                # greedy decoding
        nxt = model(img, text)[:, -1].argmax(-1, keepdim=True)
        text = torch.cat([text, nxt], 1)
    return text[:, 4:]

@torch.no_grad()
def score(model, imgs, labs, kind):
    ins, ans = zip(*[encode(l, kind) for l in labs])
    out = generate(model, imgs, torch.tensor(ins))
    ans = torch.tensor(ans)
    strict, lenient = [], []
    for o, a in zip(out.tolist(), ans.tolist()):
        o = o[:o.index(IDX["<eos>"]) + 1] if IDX["<eos>"] in o else o
        a = a[:a.index(IDX["<eos>"]) + 1]
        strict.append(o == a)
        lenient.append(all(t in o for t in a[:-1]))
    return torch.tensor(strict).float().mean().item(), torch.tensor(lenient).float().mean().item()

def most_common_pattern(model, imgs, labs, kind):
    """Most frequent answer, with values replaced by their attribute names."""
    out = generate(model, imgs, torch.tensor([encode(l, kind)[0] for l in labs]))
    name = {IDX[v]: a for a, vs in ATTR.items() for v in vs}
    eos, pats = IDX["<eos>"], []
    for o in out.tolist():
        o = o[:o.index(eos) + 1] if eos in o else o
        pats.append(" ".join(name.get(t, VOCAB[t]) for t in o))
    top = max(set(pats), key=lambda q: (pats.count(q), q))
    return f"{top} ({pats.count(top) / len(pats):.0%})"

CONDITIONS = [("caption-only", 1.0), ("instruction-only", 0.0), ("mixture 50%", 0.5)]
GROUPS = {"caption": ["describe"], "held-in": HELD_IN, "held-out pairs": HELD_OUT[:3],
          "held-out triples": HELD_OUT[3:]}
print(f"{'condition':17s}" + "".join(f"{g:>19s}" for g in GROUPS) + f"{'held-out lenient':>19s}")
for name, p in CONDITIONS:
    rows = []
    for seed in range(3):
        torch.manual_seed(seed)
        imgs, labs = make_images(3000)
        test_imgs, test_labs = make_images(600)
        model = train(imgs, labs, p, steps=1000)
        row = [sum(score(model, test_imgs, test_labs, k)[0] for k in ks) / len(ks) for ks in GROUPS.values()]
        row.append(sum(score(model, test_imgs, test_labs, k)[1] for k in HELD_OUT) / len(HELD_OUT))
        rows.append(row)
        if seed == 0:                                    # what does the model say to unseen formats?
            seen = {k: most_common_pattern(model, test_imgs, test_labs, k) for k in HELD_OUT[:2]}
    r = torch.tensor(rows)
    print(f"{name:17s}" + "".join(f"  {m:.2f} [{lo:.2f}-{hi:.2f}]" for m, lo, hi in
                                  zip(r.mean(0), r.min(0).values, r.max(0).values)))
    print(f"{'':17s}  seed 0, most frequent answers: " +
          ";  ".join(f"'{' '.join(k)} ?' -> {v}" for k, v in seen.items()))

condition                    caption            held-in     held-out pairs   held-out triples   held-out lenient
caption-only       0.93 [0.90-0.95]  0.03 [0.00-0.09]  0.18 [0.00-0.29]  0.00 [0.00-0.00]  0.37 [0.30-0.47]
                   seed 0, most frequent answers: 'side color ?' -> shape side <eos> (88%);  'shape side ?' -> shape side <eos> (86%)
instruction-only   0.00 [0.00-0.00]  0.93 [0.92-0.95]  0.02 [0.01-0.03]  0.00 [0.00-0.00]  0.29 [0.27-0.31]
                   seed 0, most frequent answers: 'side color ?' -> color side <eos> (100%);  'shape side ?' -> side <eos> (59%)
mixture 50%        0.92 [0.88-0.94]  0.96 [0.94-0.97]  0.05 [0.01-0.06]  0.00 [0.00-0.00]  0.24 [0.09-0.47]
                   seed 0, most frequent answers: 'side color ?' -> side <eos> (50%);  'shape side ?' -> shape <eos> (78%)

Each model learns the format it was trained on and little else. The caption-only model captions well but rarely answers an instruction, none of which it was trained on; the instruction-only model answers its six trained types but cannot caption at all, because describe is a token it never saw; the mixture does both, and in this toy the mixture costs neither skill anything. This is the specialization behind LLaVA-1.5’s format prompts and LLaVA-RLHF’s mixtures: a model does what its supervision shows, so every behaviour wanted at test time must be in the data.

None of the models answers the held-out orderings. The most frequent answers show why: each falls back on a format it has seen. The instruction-only model answers side color ? in the trained color side order, and the caption-only model answers both printed requests with the end of a caption, shape side. That habit happens to be exactly right for shape side ?; by our reasoning from the printed answers, it is where the caption-only model’s held-out pair score comes from, since the other two pairs start with the side, which a caption never puts first. It is an accident of the format, not an ability. This goes against what we expected before the first run, that instruction-trained models would beat caption-only training on held-out types. Lenient scoring would rank the caption-only model first on held-out types, and its ranges overlap across conditions; it measures whether the words appear, not whether the instruction was followed.

What the toy cannot show is the effect InstructBLIP measured: better answers on unseen tasks. In our reading, that effect rests on a pretrained, instruction-tuned language model (FlanT5 or Vicuna in InstructBLIP, kept frozen) that already understands the words of a new instruction and can compose them. Here every word means only what six templates taught a two-layer model trained from scratch in 1,000 steps, so a reordering it never saw is genuinely new to it. Scale, data diversity and the language model’s prior are what the example leaves out. Two changes were made after the first run: after seeing the near-zero held-out scores and running a separate diagnostic on the seed-0 models, we added the line printing each condition’s most frequent answers for the seed-0 model, to explain those scores from printed output, and removed an unused import; training, data, scoring and the table are unchanged and match the first run. Before the first run, a planned fourth condition with 10% captions was dropped and training was fixed at 1,000 steps, to keep the runtime near two and a half minutes.

Variants

• Data source. Blind text-only generation from annotations (LLaVA), templates over academic datasets (InstructBLIP), detailed captions from a seeing annotator or a captioner distilled from it (ShareGPT4V), and mixtures of all three (LLaVA-1.5, LLaVA-RLHF’s SFT+ data).
• What is tuned. The connector only (InstructBLIP), connector and language model (LLaVA, LLaVA-1.5), part of the vision encoder as well (ShareGPT4V pretraining), or low-rank adapters (LoRA) during RL (LLaVA-RLHF).
• Feedback form. Pairwise human rankings (LLaVA-RLHF), human segment corrections (RLHF-V), AI claim-level judgments (RLAIF-V); pairs built by rules, by distorting the image, editing the response or pairing models of different strength (HA-DPO, POVID and AMP-MEG, compared in RLAIF-V’s Table 1), are a further family.
• Optimizer for preferences. PPO against a learned reward with a KL penalty, offline DPO, dense DPO, or iterative DPO with fresh samples each round.

Best Practices

• State the expected output format in the instruction. A fixed format sentence per answer type let LLaVA-1.5 mix one-word VQA with long conversations (Liu et al., 2024); InstructBLIP put “short” and “briefly” in some templates for the same reason (Dai et al., 2023).
• Balance datasets explicitly when sizes differ by orders of magnitude, by square-root sampling or by repetition and subsampling, and check convergence per dataset (Dai et al., 2023; Liu et al., 2024).
• Keep conversational data in the mixture when adding short-answer datasets; LLaVA-RLHF’s single-dataset additions show how fast long-form ability can collapse (Sun et al., 2024).
• Prefer responses written by a model that sees the image, or by humans, for descriptions, and check whether the detail exceeds what the model’s input resolution can resolve (Chen et al., 2024; Liu et al., 2024).
• Make preference pairs differ only in what should be learned: segment corrections (RLHF-V) or same-model samples (RLAIF-V) rather than responses from different sources (Yu et al., 2024; Yu et al., 2025).
• Give the reward model the facts it needs, such as captions or known answers, and watch response length during RL (Sun et al., 2024).
• Evaluate on held-out tasks and on hallucination benchmarks as well as held-in benchmarks; InstructBLIP’s held-out split (Dai et al., 2023) and the benchmarks in Multimodal Hallucination are the usual choices.

Pitfalls

• Teaching the model to guess. Training targets that describe details the image does not show, whether written by a blind text model or from crops that removed the object, train the model to assert unseen content. LLaVA-RLHF’s authors name behaviour cloning on such data as a cause of hallucination (Sun et al., 2024); RLHF-V’s authors stopped using random crops for this reason (Yu et al., 2024).
• Format overfitting. Short-answer datasets without format prompts make a model answer briefly everywhere (Liu et al., 2024).
• Lenient metrics. A score that only checks whether the right words appear rewards a model that lists everything, as the example shows; benchmarks judged by language models can also reward length. LLaVA-RLHF’s PPO policy learned to write longer answers to please its reward model (Sun et al., 2024).
• Rewarding evasion. MMHal-Bench’s authors note that short or evasive responses can score well on a hallucination benchmark, a trade-off between honesty and helpfulness (Sun et al., 2024).
• Contaminated held-out sets. VQAv2 and GQA training images are in LLaVA-1.5’s mixture, so their scores are not zero-shot (Liu et al., 2024); InstructBLIP kept Visual Dialog as a held-out set despite overlapping images and said so (Dai et al., 2023).
• Licences and reproducibility. Data generated by proprietary models inherits their terms. As of November 2026, the ShareGPT4V and RLAIF-V repositories license their data CC BY-NC 4.0 for research use only, and LLaVA’s README asks users to comply with OpenAI’s terms of use for its data and with the base language models’ licences for its checkpoints (repository READMEs).

Evidence and Impact

The strongest controlled evidence that the instruction format itself matters comes from InstructBLIP’s comparison on the BLIP-2 FlanT5-XL backbone (Figure 4). Training the same model on the same datasets as plain multitask learning, without instructions or with dataset-name tags, gave held-in averages close to instruction tuning (89.0–93.7 against 93.8) but held-out averages on par with the untuned BLIP-2 (45.5–46.8 against 46.1), while instruction tuning reached 52.9 (Dai et al., 2023). The held-out average covers GQA, TextVQA, VSR, HatefulMemes, IconQA, ScienceQA, iVQA and VizWiz. On the data side, LLaVA’s ablation of its instruction types on LLaVA-Bench is reported in the LLaVA paper article: without instruction tuning the relative score was 21.5, with the full data 85.1.

Effects on hallucination run in both directions. Supervised data can add hallucination when its text claims more than the image shows, and can reduce it when it adds negative or short factual answers: in LLaVA-RLHF’s evaluation, LLaVA (7B) answered “yes” to 77.3% of POPE questions overall, LLaVA-SFT+ to 47.1% and LLaVA-RLHF to 41.8% (Sun et al., 2024, Table 7). Fine-tuning RLHF-V’s base model on VQAv2 alone lowered its MHumanEval hallucination rate from 74.7% to 65.1% (Yu et al., 2024, Table 3). LLaVA-1.5’s authors report less hallucination at higher input resolution and argue that hallucination is learned when training text describes detail finer than the model can resolve (Liu et al., 2024). Preference tuning targets hallucination directly, as the results above (RLHF-V, RLAIF-V) show on long-form benchmarks. Multimodal Hallucination explains how CHAIR, POPE, Object HalBench and MMHal-Bench measure it and compares these methods with decoding-time and post-hoc corrections.

The technique’s impact has been largely on how open models are trained. LLaVA-1.5’s authors reported the best overall results on their 12 benchmarks (11 in the abstract) for a simple model trained on 1.2M public samples in about a day on eight A100s, against models with far larger pretraining sets, and read this as questioning the belief that such models need large-scale alignment pretraining (Liu et al., 2024). Captioners distilled from a stronger annotator (ShareGPT4V) and open-model feedback (RLAIF-V) then reduced the dependence on proprietary models for both stages. In our reading, two limits remain. Most gains are reported on benchmarks judged by GPT-4 or built from the same annotation sources as the training data, and the data pipelines that depend on proprietary models cannot be rerun exactly.