Visual Instruction Tuning

“Visual Instruction Tuning” by Haotian Liu, Chunyuan Li, Qingyang Wu and Yong Jae Lee (University of Wisconsin–Madison, Microsoft Research and Columbia University) was posted to arXiv on 17 April 2023 and published at NeurIPS 2023. Its core is three pieces: a recipe for having a text-only GPT-4 write multimodal instruction data from image annotations, the LLaVA model trained on that data, and LLaVA-Bench, an evaluation in which GPT-4 grades the answers. The model family that grew out of it, with its architecture, training stages and later versions, is covered in the LLaVA article; this article covers what the paper did and reported.

The arXiv versions differ. Version 1 reported “early experiments” and scored chat quantitatively only on 30 COCO images. Version 2 (11 December 2023), whose text matches the NeurIPS paper, named that set LLaVA-Bench (COCO) and added the In-the-Wild set, its failure analysis, and appendices on broader impact, extra qualitative results and training details. Details below are from the NeurIPS version.

Problem

The paper frames two lines of work that had not met. Language-augmented vision models, from CLIP-style classifiers to captioners, each solved one task behind a fixed interface. Instruction tuning on machine-generated data had turned LLaMA into assistants such as Alpaca and Vicuna, but only for text. Multimodal models such as Flamingo, BLIP-2 and OpenFlamingo had not been tuned on vision–language instructions, and the authors note that they did worse on multimodal tasks than on text-only ones.

The authors identify the missing ingredient as data. Image–text pairs were plentiful, from Conceptual Captions to LAION, but instruction data was scarce, and collecting it from crowd workers is slow and loosely defined. The simplest conversion, a question asking for a description answered by the caption, is cheap but, in their words, lacks diversity and in-depth reasoning.

Contribution

The paper lists four contributions:

A data pipeline in which GPT-4 or ChatGPT, seeing only text, reformats image annotations into instruction data.
A large multimodal model that connects CLIP’s visual encoder to the Vicuna decoder and is fine-tuned end to end on that data, with an ensemble with GPT-4 claimed as a new state of the art on ScienceQA.
LLaVA-Bench, two evaluation sets of images, instructions and detailed annotations.
An open release of the data, code, checkpoints and a chat demo.

The authors call this the first attempt to extend instruction tuning to the language–image space.

Method

A text-only model cannot see the image, so each COCO image is passed to it as symbols: its five captions and its object bounding boxes, each a category name with normalized coordinates. Human input is limited to a few hand-written seed examples per response type, which the authors describe as the only human annotation in the pipeline. The seeds serve as in-context examples. The appendix prints the conversation prompt, whose system message tells GPT-4 to answer as if it were seeing the image, to ask about object types, counts, actions, locations and relative positions, and to include only questions with definite answers: either the content is visible, or it is confidently absent. It also asks for complex questions on background knowledge or events, again avoiding uncertain details.

The response types do not all see the boxes. The conversation prompt describes its input as five sentences, and its two seed examples, printed in the appendix, contain captions only. The detailed-description and complex-reasoning prompts, which the paper leaves to its official repository, add the boxes and tell GPT-4 to describe counts and positions in words, never mentioning coordinates or its sources. Each detailed-description request uses one of 16 phrasings, sampled at random and curated from a list GPT-4 proposed. The authors report trying both ChatGPT and GPT-4 in early experiments and finding GPT-4’s data consistently better, for example on spatial reasoning. The result, LLaVA-Instruct-158K, splits as the LLaVA article gives.

The 595K image–caption pairs for the alignment stage were chosen by noun-phrase coverage. The authors extracted noun phrases from every CC3M caption with spaCy and dropped phrases seen fewer than three times. Working from the rarest remaining phrase upward, they added its captions to the pool, keeping at most 100 per phrase. The filtered set kept 31,423 of CC3M’s 108,182 unique noun phrases. Each pair became a single-turn request, using one of 11 phrasings of “describe the image briefly”, answered with the caption.

Model and training as published

The model maps CLIP ViT-L/14 grid features into Vicuna’s word-embedding space with one trainable matrix, chosen so that data-centric experiments run quickly. Sequences follow Vicuna-v0’s chat format with ### as the stop marker; the LLaVA article gives the objective and the two training stages.

Training followed Vicuna’s hyperparameters on eight A100 GPUs: one epoch of alignment at learning rate $2 \times 10^{-3}$ (batch 128), then three epochs of instruction tuning at $2 \times 10^{-5}$ (batch 32), with Adam, no weight decay and a cosine schedule.

Results

Qualitative comparisons

The paper first reuses prompts from the GPT-4 technical report. Asked what is unusual about a photo of a man ironing clothes on a vehicle in traffic, LLaVA explains the oddity, while BLIP-2 and OpenFlamingo describe the scene. The authors stress that these images were out of domain for a model trained on about 80K unique images. The appendix adds LLaVA writing a web page from a sketch (after one fix), reading text although OCR was rare in training, and naming Elon Musk, who appears in neither training stage.

LLaVA-Bench

The evaluation builds triplets of image, ground-truth text description and question. The candidate model answers from the image. Text-only GPT-4 writes a reference answer from the text description, which the authors treat as an approximate upper bound. A text-only GPT-4 judge then sees the question, the description and both answers, weighs helpfulness, relevance, accuracy and level of detail, gives each answer an overall score from 1 to 10, and explains its scores. In the released evaluation code, the reported relative score for a question set is a ratio of mean judge scores:

\text{relative score} = 100 \times \frac{\bar{s}_{\text{model}}}{\bar{s}_{\text{GPT-4 reference}}}.

LLaVA-Bench (COCO) has 30 random COCO val2014 images, each with one question of each type made by the training-data pipeline, 90 questions in all. The authors say it studies alignment with consistent visual inputs, and they use it to ablate the training data (relative scores, all questions):

Training data	Conversation	Detail	Reasoning	All
Full LLaVA-Instruct-158K	83.1	75.3	96.5	85.1
Detail + complex reasoning only	81.5	73.3	90.8	81.9
Conversation + 5% detail + 10% reasoning	81.0	68.4	91.5	80.5
Conversation only	76.5	59.8	84.9	73.8
No instruction tuning	22.0	24.0	18.5	21.5

By the authors’ reading, instruction tuning adds over 50 points, and a little description and reasoning data adds about 7 points overall, raising conversation scores too.

LLaVA-Bench (In-the-Wild) has 24 images with 60 questions: indoor and outdoor scenes, memes, paintings and sketches. Each image comes with a detailed, manually curated description that the judge and the reference use.

Model	Conversation	Detail	Reasoning	All
OpenFlamingo	19.3 ± 0.5	19.0 ± 0.5	19.1 ± 0.7	19.1 ± 0.4
BLIP-2	54.6 ± 1.4	29.1 ± 1.2	32.9 ± 0.7	38.1 ± 1.0
LLaVA	57.3 ± 1.9	52.5 ± 6.3	81.7 ± 1.8	67.3 ± 2.0

The spreads come from three inference runs. Judging one fixed set of LLaVA outputs three times gave 66.7 ± 0.3, which the authors take as evidence that the GPT-4 judge is consistent. In our reading, the larger spread across runs comes mostly from decoding, especially for detailed descriptions.

ScienceQA

ScienceQA (Lu et al., 2022) has about 21K multiple-choice science questions across 3 subjects, 26 topics, 127 categories and 379 skills, split 12,726 / 4,241 / 4,241. LLaVA was fine-tuned for 12 epochs as a single-turn conversation that outputs reasoning and then the answer. It reached 90.92% test accuracy, against 91.68% for MM-CoT Large, 88.40% for humans and 75.17% for GPT-3.5 with chain of thought. On questions with image context it scored 88.00%.

Text-only GPT-4 with two in-context examples scored 82.69%. It often failed by saying that it lacked the image or plot. The authors combined the two models in two ways. Using LLaVA’s answer whenever GPT-4 gave none reached 90.97%, almost the same as LLaVA alone. When the two disagreed, GPT-4 was prompted again with the question and both answers and asked for a final answer. This judge scheme reached 92.53% and improved every question category, a new state of the art. The authors say it was, to their knowledge, the first use of GPT-4 for model ensembling. They explain that it helped even on image-context questions because some of them do not need the image. In the appendix example, LLaVA claimed that a rocking chair was made of wood and silk, and the judge, unable to see the image, chose wood on commonsense grounds.

The ablations, all on ScienceQA test accuracy, isolate four choices:

Variant	Accuracy (%)
Best (13B, features before CLIP’s last layer, reasoning first)	90.92
Last-layer CLIP features	89.96
Answer before reasoning (listed under last-layer features)	89.77
No alignment pretraining	85.81
7B language model	89.84

Reasoning-first training reached 89.77% after 6 epochs, where answer-first needed 12. Training for 24 epochs did not help. The authors conclude that chain-of-thought ordering mainly speeds convergence. They read the 5.11-point drop without pretraining as showing that the alignment stage matters.

Impact

The open release made the recipe easy to reuse. The authors’ follow-up, LLaVA-1.5 (Liu et al., 2024), kept the design and LLaVA-Bench (In-the-Wild) among its 12 benchmarks. It was evaluated there with GPT-4 as the judge, scoring 72.5 for the 13B model. LLaVA-Med (Li et al., 2023) moved the recipe to biomedicine. It was initialized from LLaVA, trained on instructions that language-only GPT-4 wrote from PubMed Central figure captions, and evaluated with the same relative-score judging.

Limitations

LLaVA-Bench (In-the-Wild) was built to expose weaknesses, and the authors discuss them. Naming a ramen restaurant requires broad knowledge and multilingual reading, and naming a yogurt brand requires high-resolution input. Asked whether a fridge held strawberry-flavored yogurt, LLaVA said yes when the fridge held yogurt and strawberries. The authors read this failure as perceiving the image as a “bag of patches”. The broader-impact appendix adds hallucination (with a warning about medical uses), biases inherited from CLIP and LLaMA/Vicuna, and the evaluation’s blindness to visual hallucination and fine-grained understanding. The authors found the GPT-4 judge consistent and accurate in their study but left its robustness to future work.

Later work and the paper’s own design point to further limits:

Short answers and “yes”. The NeurIPS version sends readers to LLaVA-1.5 for academic benchmark results. LLaVA-1.5’s authors write that the original model falls short on academic benchmarks that expect short answers, such as single words, and tends to answer “yes” to yes/no questions. They attribute both to the lack of such data in its training. Visual question answering and Multimodal Hallucination cover these benchmarks.
A judge without the image. In our reading, LLaVA-Bench rewards answers that agree with a text description, so details the description leaves out cannot earn credit, and the reference comes from the same model that wrote the training data. The COCO set is also generated by the training pipeline itself. The two sets have only 90 and 60 questions.
A proprietary dependency. In our reading, because the training data, the judge and the best ScienceQA result all depend on GPT-4, none can be rerun exactly with the same teacher.

What Came After

LLaVA-1.5 replaced the linear projection with a two-layer MLP, raised the input resolution to 336 pixels, and added academic VQA data with response-format prompts that ask for short answers. The LLaVA article compares it with LLaVA-NeXT, LLaVA-OneVision and LLaVA-Med, and the vision–language model article follows the encoder–projector–LLM design more broadly.