VQA: Visual Question Answering

“VQA: Visual Question Answering” is a paper by Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Dhruv Batra and Devi Parikh (Virginia Tech) and Margaret Mitchell and C. Lawrence Zitnick (Microsoft Research), presented at the IEEE International Conference on Computer Vision (ICCV) in 2015, with Antol and Agrawal marked as equal first authors. It proposed visual question answering in its free-form, open-ended form: given an image and a natural-language question about it, produce a natural-language answer. It also introduced a dataset of about 0.76 million questions about 204,721 COCO photographs and 50,000 clipart scenes, each question answered by ten crowd workers, and an accuracy measure that scores an answer by how many of them gave it. This article covers the publication. The dataset as released and used today, the later VQA v2, the metric as the official evaluation code computes it, the challenges and the milestones are described in the VQA dataset article.

The paper exists in several versions whose content differs, so each detail below names its version. This article was written from the nine-page ICCV paper in the Computer Vision Foundation’s open-access copy and from all seven arXiv versions of 1505.00468 (3 May 2015 to 27 October 2016). Version 3 (15 October 2015) has the ICCV main text, apart from layout, followed by appendices that appear to correspond to the supplementary material the ICCV paper refers to; the open-access page lists no supplementary file. Version 7 is an extended version with Agrawal, Lu and Antol as equal first authors. A journal version appeared in the International Journal of Computer Vision (IJCV, volume 123, 2017; online 8 November 2016). It could not be read for this article: the publisher’s page offers it for purchase and no open copy was found. Its relation to v7, posted twelve days earlier, is therefore not verified.

Problem

The authors’ starting point was image and video captioning, research combining computer vision, natural language processing and knowledge representation that had grown sharply in the preceding year. They argued that it fell short as an “AI-complete” task: a coarse, scene-level understanding of an image combined with word n-gram statistics already produced reasonable captions, and automatic evaluation of captions was still an open research problem. A task worth pursuing, in their view, should need knowledge from more than one field and have a well-defined quantitative metric. Questions about images met both conditions. A question asks for specific information, often in the background or the context of a scene. And since many answers are one to three words long, or can be offered as multiple choice, an algorithm can be scored by counting the questions it answers correctly.

Earlier question-answering work on images, as the paper describes it (§2), used restricted and small settings:

• Malinowski and Fritz (2014) answered questions about 1,449 images whose answers came from a closed world of 16 basic colours or 894 object categories.
• The visual Turing test of Geman et al. (2015) generated questions from templates over a fixed vocabulary of objects, attributes and relationships, on 2,591 images.
• VizWiz (Bigham et al., 2010) had crowd workers answer questions that blind users asked about photos they took; the paper names helping the visually impaired as a motivating application.

The authors listed as concurrent work an LSTM-and-CNN answer generator by Malinowski, Rohrbach and Fritz (ICCV 2015), a model similar to one the paper evaluates. They also named question–answer pairs collected in Chinese for COCO images by Gao et al. and questions generated automatically from COCO captions by Ren et al. Against these, the VQA dataset was to be two orders of magnitude larger, with questions and answers written by people.

Contribution

• The task, in two forms: open-ended answering (called “open-answer” in the ICCV text and “open-ended” from arXiv v5 on) and multiple choice among 18 candidates.
• A dataset of three questions for each of 204,721 COCO images and 50,000 newly made abstract scenes, each question answered by ten workers with a self-rated confidence, plus three answers per question from workers who did not see the image.
• An accuracy measure that gives full credit to an answer at least three workers gave, with a fixed normalization of answer strings.
• An analysis of question types, answer lengths and answer distributions, of human accuracy with and without the image, of how much commonsense the questions need, and of how questions differ from captions.
• Baselines, from answer priors to a multilayer perceptron on bag-of-words and VGGNet features and an LSTM model, compared with human accuracy per question type.
• A plan for an evaluation server and an annual challenge with a workshop.

Method

Images and abstract scenes

The real images were COCO’s 123,287 training and validation images and its 81,434 test images, chosen because COCO had been collected to find scenes with several objects and rich context (§3, citing Lin et al., 2014). The abstract scenes were new: 50,000 clipart scenes, more realistic than those of the authors’ earlier abstract-scene work, with 20 “paperdoll” people of different genders, races and ages, 8 expressions and adjustable limbs, more than 100 objects and 31 animals. The stated purpose was to let researchers study the high-level reasoning VQA needs without first having to recognize objects in noisy real photographs. The arXiv appendices add that workers composed scenes by dragging, flipping and scaling clipart, that most animals had five discrete poses, and that scale set the rendering order (Appendix IV of v3, Appendix VI of v7).

Questions

The authors wanted questions that need the image, ranging from low-level ones (“what color is the cat?”) to ones that need commonsense about the scene, but not ones answerable from commonsense alone. In pilot studies, workers were asked for questions that a toddler, an alien or a smart robot would have trouble answering, and the smart-robot framing gave the most interesting and diverse questions. The final interface described a robot that can already name and locate objects and recognize scenes, expressions, poses and object properties, and told workers to “stump this smart robot!”. It also asked for questions that need the image. Three workers wrote one question each per image or scene, each seeing the questions already asked for it; the arXiv appendix adds that workers were told the robot already knew the answers to those earlier questions (Appendix V of v3, Appendix VII of v7).

Answers

Because a question can have several correct answers (“white”, “tan”, “off-white”) and people can disagree, ten different workers answered each question, none of them its author. They were told to give “a brief phrase and not a complete sentence” and to avoid conversational language and opinion. Each also answered whether they thought they had answered correctly, with “no”, “maybe” or “yes”. Three further workers answered each question without seeing the image. Their answers served both as plausible wrong options for multiple choice and as a measure of how far commonsense alone goes.

Scoring

For the open-ended task the ICCV paper defines the accuracy of an answer $a$ as

$$\mathrm{Acc}(a) = \min\left(\frac{\#\,\text{humans that provided } a}{3},\ 1\right),$$

so an answer is fully correct when at least three workers gave exactly that string. A footnote adds that machine accuracy is averaged over all $\binom{10}{9}$ sets of nine annotators, to be consistent with the human accuracies reported in §4. The paper does not otherwise describe how human accuracy is computed. The text states that before comparison all answers are lowercased, number words converted to digits, and punctuation and articles removed. It rejects soft matching with word embeddings such as Word2Vec, since they group words that must stay distinct, such as “left” and “right”. Version 7 adds a reason for avoiding BLEU and ROUGE: with 89.32% of answers a single word, there are no higher-order n-gram matches, and low-order ones reduce to exact matching.

The released evaluation code does not follow this description exactly. Its current version applies the normalization only when the ten human answers are not all identical, and before 2021 it treated predictions and human answers differently. The VQA dataset article describes the code line by line and shows the effect.

For multiple choice, the 18 candidates combine the most common of the ten answers, the no-image answers (padded from bag-of-words nearest-neighbour questions when fewer than three distinct ones exist), the ten most popular answers in the dataset and correct answers to random questions. A chosen option is scored with the same formula. Including the popular answers, the authors explain, stops an algorithm from inferring the question type from the options, for instance a yes/no question from the presence of “yes” and “no”. Version 7 notes that since the ten answers can differ, more than one option can earn credit.

Splits

The real images follow COCO’s splits, including the test-dev, test-standard, test-challenge and test-reserve subsets of the test images; the abstract scenes are split 20,000 / 10,000 / 20,000 for training, validation and test. Version 7 explains the roles: test-dev for debugging and validation experiments, test-standard for comparisons in papers and a public leaderboard, test-reserve hidden to detect overfitting, and test-challenge to decide the winners. It also says that abstract scenes have no subsplits and that test-dev “allows for unlimited submission”. The 2016 challenge page, as of October 2026, says instead that test-dev allows 10 submissions per day.

Results

What the data showed

The analysis in §4 uses the training data, but the section opens with counts for the whole dataset: 614,163 questions with 7,984,119 answers for the COCO images and 150,000 questions with 1,950,000 answers for the abstract scenes, both totals including the no-image answers (13 per question).

• Questions. Grouped by their first words, real-image and abstract-scene questions have similar distributions (Fig. 3), which the authors took as evidence that the clipart elicits the same kinds of questions. Most questions are four to ten words long (Fig. 4).
• Answers. One-word answers make up 89.32% of real-image answers and 90.51% of abstract-scene answers, two-word answers 6.91% and 5.89%, and three-word answers 2.74% and 2.49%. The authors argued that this brevity makes automatic evaluation feasible, while the questions still need reasoning to reach these short answers. There were 23,234 distinct one-word answers for real images and 3,770 for abstract scenes.
• Yes/no and number questions. Questions answered “yes” or “no” (or sometimes “maybe”) are 38.37% of real-image and 40.66% of abstract-scene questions, with “yes” given in 58.83% and 55.86% of their answers. Number questions are 12.31% and 14.48%, and “2” makes up 26.04% and 39.85% of their answers.
• Agreement. Agreement between workers grew with their self-rated confidence, but even confident workers often differed, for instance with near-synonyms such as “happy” and “joyful” (Fig. 6). Questions had on average 2.70 distinct answers on real images and 2.39 on abstract scenes.

The human studies are collected in Table 1, which reports test-standard accuracy of people answering free-form, with the caption rows measured on 3,000 training questions about 1,000 images:

Data	People shown	All	Yes/no	Number	Other
Real images	question only	40.81	67.60	25.77	21.22
Real images	question and a caption	57.47	78.97	39.68	44.41
Real images	question and image	83.30	95.77	83.39	72.67
Abstract scenes	question only	43.27	66.65	28.52	23.66
Abstract scenes	question and a caption	54.34	74.70	41.19	40.18
Abstract scenes	question and image	87.49	95.96	95.04	75.33

The authors read the first rows as showing that commonsense alone is not enough: without the image, people answer “other” questions correctly only about 21% of the time. They read the caption rows as showing that generic captions miss what the questions ask about. A Kolmogorov–Smirnov test found the distributions of nouns, verbs and adjectives in captions significantly different from those in questions and answers ($p < .001$) for both image types. Agreement with the image, 83.30% on real images, was above 95% on yes/no questions and below 76% on the others. The authors attributed this partly to exact string matching, which ignores synonyms and plurals, and noted that deciding automatically which answers are synonyms is hard because the right granularity varies between questions.

A further study on 10,000 real-image training and validation questions asked ten workers per question whether it needed knowledge external to the image, and which age group could first answer it (§4.3). For 47.43% of the questions at least three of the ten said commonsense was needed, and for 18.14% at least six. The age votes were spread as 15.3% toddler (3–4), 39.7% younger child (5–8), 28.4% older child (9–12), 11.2% teenager (13–17) and 5.5% adult (18+). The two resulting rankings of questions correlated at 0.58 (the paper calls this Pearson’s rank correlation). From v5 the paper notes that the ages are those perceived by workers, and v7 adds that the ordering of question types therefore matters more than the absolute ages. Versions 6 and 7 also give the average perceived age, 8.92 years, and the average degree of commonsense, 31.01%.

Baselines

All baselines were trained on training plus validation data and evaluated on test-dev, except those using human captions, which were trained on training data and evaluated on validation data (§5). Human accuracies were on test-standard. As reference points, the ICCV text gives 0.12% on test-standard for a random answer from the 1,000 most frequent ones, 29.72% for always answering “yes”, 36.18% for the most popular answer per question type and 40.61% for a nearest-neighbour method on validation data. The sentence leaves the split of the middle two numbers unstated; the baseline appendix added in v4 places 36.18% on validation data.

The learned models chose among the K = 1,000 most frequent answers, which cover 82.67% of the training and validation answers:

• Multilayer perceptron. A network with two hidden layers of 1,000 tanh units and dropout 0.5. The question was a 1,030-dimensional bag of words: the 1,000 most frequent question words plus 30 dimensions for the ten most frequent first, second and third words. The image was the 4,096-dimensional last hidden layer of VGGNet, and a caption a bag of its 1,000 most frequent words. Combinations concatenated the first hidden layers of networks trained on each input.
• LSTM. An LSTM read one-hot question words. The image features were linearly projected to 1,024 dimensions, multiplied element-wise with the question encoding, and a softmax over the answers followed.

Open-ended answering took the highest-scoring of the 1,000 answers, and multiple choice the highest-scoring of the 18 candidates. Table 2 of the ICCV paper (test-dev, real images; caption rows on validation data) gives, for all questions:

Model	Open-ended	Multiple choice
Question (bag of words)	48.09	53.68
Image	28.13	30.53
Question + image	52.64	58.97
LSTM question	48.76	54.75
LSTM question + image	53.74	57.17
Caption (validation)	26.70	28.29
Question + caption (validation)	54.70	59.85

The text also names a question + image + caption combination, but the table has no row for it. The authors read the roughly 48% of the question-only models as showing that the type of question is informative of the answer. All models fell far short of people. Per question type (Table 3), image features added nothing for “is the” questions (70.76% from the question alone, 69.87% with the image) or “how many” questions (43.78% and 40.33%). They helped where scene-level information suffices: “what sport” questions rose from 37.96% to 81.12%. From v5, the caption of Table 3 states that its machine columns use the bag-of-words question model. The LSTM question + image model, chosen on test-dev, scored 54.06% on test-standard, against 83.30% for people.

The paper closes its results with the estimate that this model “performs as well as a 4.45 year old child”. Appendix V of v4 describes the computation: on the validation questions with age annotations, a weighted average of each question’s average perceived age, weighted by the model’s accuracy on that question. In our reading, the reported numbers cannot be a weighted average in the usual sense, divided by the sum of the weights. From v5 the paper gives 4.74 years for its later model; v6 and v7 add an average of 8.98 over the same questions, and their Fig. 11 shows the model’s accuracy falling only from 61.07% for the youngest group to 47.83% for adults. With accuracy this similar across age groups, an average weighted by it would stay close to 8.98, nowhere near 4.74. The figures instead match the sum divided by the number of questions, which is close to the model’s accuracy times the average age when accuracy varies little with age: 4.74 / 8.98 ≈ 0.53, close to the 54.40% that Goyal et al. (2017) later reported for this model on v1 validation data. The same holds for the “degree of commonsense” in v6 and v7: 17.35% against an average of 31.23% (ratio 0.56). Read this way, the “age” is the model’s accuracy on a rescaled axis.

How the Paper Changed

The arXiv versions record the dataset and the baselines while they were being built:

Version	Date	What changed
v1	3 May 2015	Statistics and baselines from a partial collection with three answers per question; the score was the percentage of human answers matching; baselines were a one-hidden-layer network with 50 units on AlexNet’s 1,000 softmax outputs, trained on 30,000 images and tested on 20,000
v2	16 June 2015	Ten answers per question being collected; statistics still from the 20 April 2015 data, with the same score
v3	15 October 2015	The ICCV paper: full dataset, min(#/3, 1) score, MLP and LSTM baselines, 54.06%
v4	18 November 2015	Appendices on the baselines (Skip-Thought nearest neighbours) and the age estimate; the human multiple-choice study, which v1–v3 report from five answers per question on 3,000 real-image questions, redone with three answers on 3,000 questions per image type
v5	7 March 2016	The “deeper LSTM Q + norm I” model: 58.16% open-ended and 63.09% multiple choice on test-standard; first workshop announced for CVPR 2016; age estimate 4.74 years
v6	20 April 2016	Author order changed to Agrawal, Lu, Antol, Mitchell, Zitnick, Batra, Parikh; comparison with challenge leaderboard entries; average age and commonsense figures and model accuracy per age group (Fig. 11)
v7	27 October 2016	Filtered-data and other ablations, the BLEU and ROUGE argument, the split roles, leaderboard comparison dated 28 October 2016

Under v1’s score, human accuracy with the image was 61.58% on real images, and the question-only network reached about 27%. The same rows appear under the new score as 83.30% and about 48% in the ICCV paper, with ten answers per question and different data, so the two sets of numbers are not comparable.

The improved model of v5–v7 encodes the question with a two-layer LSTM, keeping the last cell and hidden states (2 × 2 × 512 = 2,048 values) and mapping them to 1,024 dimensions. It $\ell_2$-normalizes the VGGNet features before projecting them to 1,024 dimensions, fuses the two by element-wise multiplication, and classifies with the two-layer perceptron. VGGNet stays frozen and the rest is trained end to end with cross-entropy (§5.2 of v7). On test-dev it scored 57.75% open-ended and 62.70% multiple choice (Table 2 of v7). In the same table, the reference baselines are 29.66% for “yes”, 37.54% for the per-type prior and 42.70% for nearest neighbours, all on test-dev, replacing the ICCV figures.

Version 7’s ablations (Table 4, test-dev) changed one element at a time:

• Concatenation instead of multiplication: 0.95 points lower.
• Unnormalized image features: 0.16 lower.
• 500 answers instead of 1,000: 0.82 lower.
• Training only on answers with confidence “yes”, for questions where at least five of the ten answers had it: 1.13 lower. This filter removed 6.83% of the 369,861 training and validation questions, leaving 8.77 answers per question on average.
• 2,000 answers: 0.40 higher.
• A question vocabulary truncated to words seen at least five times: 0.24 higher; at least eleven times: 0.06 higher.

Three variants thus scored higher on test-dev than the model the paper reports as its best, which was kept. Table 5 compares its test-standard scores with leaderboard entries “as of October 28, 2016”, led by snubi-naverlabs at 60.60% open-ended and 64.95% multiple choice. The same rows already appear in v6, of April 2016, whose PDF as arXiv serves it dates them “as of December 7, 2024”. In our reading both dates are the dates the PDFs were compiled, not that of the snapshot, which is no later than April 2016.

Impact

The paper turned visual question answering into a benchmark task. The evaluation server and annual challenge it promised followed, with the first workshop at CVPR 2016 (v7), and challenges ran until 2021 (see the VQA dataset article). Multimodal compact bilinear pooling (Fukui et al., 2016) won the 2016 real-image track (Goyal et al., 2017). The dataset’s format — open-ended crowd questions on COCO images, ten answers each and the consensus score — was kept by VQA v2 (Goyal et al., 2017), on which the 2017 to 2021 challenges ran. The format also carried over to other data: VizWiz (Gurari et al., 2018) collected ten answers per question from crowd workers by modifying the VQA protocol, kept the “brief phrase” instruction, and reported the VQA accuracy alongside caption metrics. This was the kind of task-specific collection, from visually impaired users, that the ICCV conclusion had suggested. The paper’s deeper LSTM Q + norm I model served as a baseline in later analyses; Goyal et al. used it as the model introduced with the dataset.

The PAMI Technical Committee awarded its 2025 Mark Everingham Prize to “The VQA Series of Challenges”, naming A. Agrawal, Y. Goyal, A. Shrivastava, D. Batra, D. Parikh and collaborators. The prize recognizes the challenge series, not the paper. The VQA dataset article records the challenges and the results that followed.

Limitations

• Exact matching. The authors acknowledged that string matching ignores synonyms and plurals and lowers measured agreement on non-yes/no questions. The text describes the normalization without the conditions the released code applies, so the paper alone does not define the score that servers report.
• Language priors. The paper’s own numbers show how much the question alone gives: about 48% for the question-only models against 54% with the image. The authors read this as the question type being informative. Later work traced it to biases in the data. Goyal et al. (2017), citing Zhang et al. (2016), describe a visual priming bias: workers wrote questions while looking at the image, so a question such as “is there a clock tower in the picture?” is asked only about images that contain one. Goyal et al. (2017) found that “tennis” answers 41% of “what sport is” questions and that blindly answering “yes” to questions starting “do you see a” scores 87%. Trained on v1 training data, the paper’s deeper LSTM Q + norm I model fell from 54.40% on v1 validation data to 47.56% on balanced validation data (Goyal et al., 2017, Table 1).
• Mixed evaluation sets. Human accuracies are on test-standard, machine accuracies on test-dev, caption experiments on validation data, and the human caption study on 3,000 training questions. Comparisons between these rows cross splits, and the best model was selected on the test-dev server.
• Age and commonsense estimates. As computed (above), they are rescaled accuracies rather than ages, and the ages themselves are workers’ perceptions, as v7 says.
• Not task-specific. The authors noted that their general questions might not serve particular applications, and suggested collecting questions from visually impaired users or within single domains.

What Came After

The topics the paper raised were each taken up, mostly by its authors and their groups:

• Balanced data. Zhang et al. (2016) paired binary questions on abstract scenes with two scenes that give opposite answers. Goyal et al. (2017) did the same for real images in VQA v2, adding for most questions a similar image with a different answer.
• Changing priors. VQA-CP (Agrawal et al., 2018) re-split the data so that answer distributions per question type differ between training and test.
• Task-specific questions. VizWiz (Gurari et al., 2018) built a dataset from questions asked by blind people, as the conclusion proposed.
• Models. The element-wise fusion of a CNN image vector and an LSTM question vector gave way to attention, bilinear pooling and then pretrained vision–language transformers; visual question answering describes that progression.