VQA Dataset

The VQA dataset is a collection of open-ended questions about images, each answered by ten crowd workers, built for visual question answering. Antol, Agrawal, Lu, Mitchell, Batra, Zitnick and Parikh (Virginia Tech and Microsoft Research) introduced it at ICCV 2015. The first full release, VQA v1.0 (October 2015), has three questions for each of 204,721 COCO images and 50,000 abstract clipart scenes. VQA v2.0 (Goyal et al., CVPR 2017; full release April 2017) adds, for most questions, a similar image on which the same question has a different answer, so that the question alone predicts the answer less well. Both versions are scored with a consensus accuracy against the ten human answers through evaluation servers, and six yearly challenges ran from 2016 to 2021. The PAMI Technical Committee gave “The VQA Series of Challenges” the 2025 Mark Everingham Prize, “for stimulating a new strand of vision and language research”. The versions are also written VQA 1.0, VQA v2, VQAv2 or VQA 2.0; the official site calls v2 “Balanced Real Images”.

Purpose

Antol et al. (2015) proposed the task as a test of image understanding that, unlike captioning, can be scored automatically: many questions about a picture have one-to-three-word answers, or can be posed as multiple choice. Workers were asked to write questions that a “smart robot” able to name objects and scenes would still get wrong, and that need the image to answer. The abstract scenes were added so that researchers interested in high-level reasoning could work without noisy recognition of real photographs.

VQA v2 had a narrower goal. Goyal et al. (2017) set out to counter the language priors of v1, the regularities that let a model answer from the question alone, so that progress on the benchmark would reflect image understanding.

Dataset Size

The official home page lists each release (as of October 2026). “Plausible” answers were given by workers who saw the question but not the image; they feed the multiple-choice candidates.

Release (home page)	Images	Questions	Ground-truth answers
Beta v0.1, June 2015	10,000 COCO (from train)	30,000	300,000
Beta v0.9, July 2015	123,287 COCO (train and val)	369,861	3,698,610
v1.0 real images, October 2015	204,721 COCO (train, val, test)	614,163	6,141,630, plus 1,842,489 plausible
v1.0 abstract scenes, October 2015	50,000 scenes	150,000	1,500,000, plus 450,000 plausible and 250,000 captions
Beta v1.9 balanced real images, March 2017	123,287 COCO (train and val)	658,111	6,581,110, plus 1,974,333 plausible
Balanced binary abstract scenes, March 2017	31,325 scenes (train and val)	33,383	333,830
v2.0 balanced real images, April 2017	204,721 COCO (train, val, test)	1,105,904	11,059,040

The papers count differently. Antol et al. (ICCV version) give 7,984,119 answers for the real images and 1,950,000 for the abstract scenes, which include the three answers per question given without the image (13 per question). Goyal et al. describe v2 as about 1.1 million image–question pairs with about 13 million answers, and give more than 443K train, 214K val and 453K test pairs, where the download page lists 447,793 test questions. The 2017 and 2021 challenge pages describe v2 as “more than 250K images”, which matches the v1 total with abstract scenes rather than the 204,721 COCO images listed for v2. The home page’s own headline, “265,016 images (COCO and abstract scenes)”, matches no release total; in our arithmetic it equals 204,721 + 50,000 + 10,295, the last being the number of complementary training scenes Zhang et al. (2016) report, but the page does not explain it.

Questions and Answers

There are no object classes. Questions are grouped instead by question type, the first words of the question, and answers by answer type: “yes/no”, “number” or “other” (download page). The repository lists 65 question types for real images (64 prefixes such as “how many”, “what color is the” and “is there a”, plus “none of the above”, added in April 2017) and 81 for abstract scenes; a question takes the longest prefix that matches, and the types are unchanged from v1 to v2 (README at commit a013f00). Both fields are stored in each annotation, so per-type accuracies need no parsing at evaluation time.

Antol et al. analysed the v1 training set. Most questions have four to ten words. Answers are short: 89.32% of the real-image answers are one word, 6.91% two words and 2.74% three. On real images, 38.37% of questions are answered “yes” or “no” (sometimes “maybe”), and 58.83% of those answers are “yes”; 12.31% are number questions, and “2” makes up 26.04% of their answers. Questions have on average 2.70 distinct answers among the ten (2.39 for abstract scenes). The 1,000 most frequent answers cover 82.67% of the training and validation answers, which is why early models classified over them.

Balancing changed these distributions. In v2, binary questions are much closer to an even split of “yes” and “no”, “baseball” overtakes “tennis” for “what sport” questions, and the entropy of the answer distribution per question type, weighted by type frequency, is 56% higher than in v1 (Goyal et al., 2017).

Annotation Types

Questions. Three workers wrote one question each per image or scene, seeing the questions already asked so as to vary them (Antol et al., 2015).

Answers. Ten different workers answered each question, none of them its author, and were asked for “a brief phrase and not a complete sentence”. Each also said whether they thought they had answered correctly (“no”, “maybe” or “yes”), stored as answer_confidence. Three more workers answered without seeing the image; these plausible answers are kept for the multiple-choice task. Each annotation also carries multiple_choice_answer, the most frequent of the ten answers (download page).

Post-processing. Before release, questions and answers were spell-corrected with Bing Speller, questions were capitalized and given a final “?”, and answers were lowercased, stripped of periods (except decimal points) and articles, had number words turned into digits and missing contraction apostrophes added (download page). The released answers are therefore already close to the form the evaluation code compares.

Abstract scenes. The clipart contains 20 “paperdoll” human models of varying gender, race and age with 8 expressions and adjustable limbs, over 100 objects and 31 animals (Antol et al., 2015). Five captions were collected per scene (released for train and val), and each scene has a composition file giving the position and orientation of every clipart object, so scenes can be re-rendered (download page).

Complementary images (v2). For each (image, question, answer) triple of v1 real images, a worker was shown the 24 nearest neighbours of the image in VGGNet fc7 feature space and asked to pick one on which the question still makes sense but the answer differs; ten new workers then answered the question on the chosen image. Workers chose “not possible” for 22% of the questions, about 135K, and for about 9% of the questions the majority answer on the new image equals the original one. Goyal et al. collected about 195K complementary images for train, 93K for val and 191K for test, and added about 18K further test pairs “to detect anomalous trends”. The download page provides the pair lists: 200,394 pairs in train and 95,144 in val.

Splits

Real images follow COCO: train2014 (82,783 images), val2014 (40,504) and test2015 (81,434), in v1 and v2 alike.

Split	v1 real questions	v2 questions	v1 abstract scenes (questions)
train	248,349	443,757	20,000 (60,000)
val	121,512	214,354	10,000 (30,000)
test	244,302	447,793	20,000 (60,000)

Counts from the download pages and the repository README. Train and validation answers are public; test answers are not.

The real-image test set is divided, as in COCO, into test-dev, test-standard, test-challenge and test-reserve. For v1, test-dev uses the same images as COCO’s test-dev split, and the other three are roughly the size of COCO’s but may contain different images (2016 challenge page); Fukui et al. (2016) describe test-dev as a 25% subset of test. Abstract scenes have a single test set, without these subsplits.

The balanced binary abstract scenes (Zhang et al., 2016) have only train (20,629 scenes, 22,055 questions) and val (10,696 scenes, 11,328 questions) on the download page; the paper calls the 11,328-question set its balanced test set.

Tasks

• Open-ended answering: produce a free-form answer string. This is the only task in v2.
• Multiple choice (v1 only): pick one of 18 candidates per question, built from the most common of the ten answers, the plausible no-image answers (padded from nearest-neighbour questions), the ten most popular answers in the dataset (“yes”, “no”, “2”, “1”, “white”, …) and correct answers to random questions, in random order (Antol et al., 2015). A chosen option is scored with the same consensus accuracy, so more than one option can earn credit (extended arXiv version, v7).
• Binary yes/no on balanced abstract scenes (Zhang et al., 2016), where nearly every question comes with two scenes, one answered “yes” and one “no” (about 6% of the test questions could not be balanced).

The 2016 challenge ran four tracks, open-ended and multiple choice on real images and on abstract scenes; the 2017–2021 challenges ran open-ended answering on v2 real images only. Visual Question Answering describes the task, its models and the benchmarks that followed.

Evaluation

The metric as stated

The evaluation page and the paper give the accuracy of a predicted answer $a$ as

$$\mathrm{Acc}(a) = \min\left(\frac{\#\,\text{humans that said } a}{3},\ 1\right),$$

so an answer counts as fully correct when at least three of the ten workers gave it. A footnote in the paper and a line on the page add that, “to be consistent with human accuracies”, machine accuracy is averaged over all $\binom{10}{9}$ sets of nine annotators. The reason, in our reading, is that a worker’s own answer cannot count towards its agreement with the others, so human accuracy is computed against nine answers, and the machine is scored the same way. Antol et al. reject soft matching with word embeddings because it would merge answers such as “left” and “right”.

The metric as the official code computes it

The official evaluation is vqaEval.py in the GT-Vision-Lab/VQA repository, written for Python 2.7; the current version is commit a013f00 (11 March 2021). For each question it does the following.

1. Replace newlines and tabs with spaces and strip surrounding whitespace, in the prediction and in the ten human answers.
2. Only if the ten human answers are not all identical, normalize the human answers and the prediction:
   • Punctuation: for each of ; / [ ] " { } ( ) = + \ _ - > < @ ` , ? !, delete it if the original string has that character next to a space, or if the string contains a comma between two digits anywhere; otherwise replace it with a space. Apostrophes and colons are kept, so “girl’s” and “10:30” survive. Then delete every period not followed by a digit (the regular expression (?!<=\d)(\.)(?!\d) was presumably meant as a lookbehind, but as written only the lookahead acts; the call also passes re.UNICODE where the replacement count goes, which caps it at 32 periods), so “3.5” is kept and “5.” becomes “5”.
   • Digits and articles: lowercase, split on whitespace, map the words “none”, “zero”, “one” to “ten” to digits, drop “a”, “an” and “the”, and add missing apostrophes to contractions from a fixed list (“dont” becomes “don’t”).
3. Count, with exact string equality, how many human answers equal the prediction. For each of the ten annotators $k$, score $\min(1, m_k/3)$, where $m_k$ is the number of matches among the other nine, and average the ten scores:

$$\mathrm{Acc}(a) = \frac{1}{10} \sum_{k=1}^{10} \min\left(1,\ \frac{\big|\{\, j \neq k : h_j = a \,\}\big|}{3}\right).$$

With $m$ matching annotators this is $\tfrac{1}{10}\big[m \min(1, \tfrac{m-1}{3}) + (10 - m) \min(1, \tfrac{m}{3})\big]$, which gives 0, 0.3, 0.6, 0.9 and 1 for $m = 0, 1, 2, 3$ and $m \geq 4$. Three matching annotators are not enough for full credit.

4. The overall score is the mean over all questions, each counting once, times 100 and rounded to two decimals; the same mean is reported per question type and per answer type, taken from the annotation file. The result loader (loadRes in vqa.py) refuses a file that does not answer every question in the annotation set or that answers a question outside it.

Three consequences follow from the code. Normalization is conditional: when all ten workers gave the same string, the prediction must match it exactly, so “Yes” scores 0 against ten “yes” answers, while it scores 1 if a single worker said “no”. The rules have side effects: a hyphen becomes a space (“t-shirt” → “t shirt”) unless a comma between digits appears elsewhere in the string (“2,000 t-shirts” → “2000 tshirts”), and “one-way” becomes “1 way”. And the metric is literal: “there are 2 circles” does not match “2”.

The 2021 change. Until March 2021 (commit 458a8b2 of October 2016, whose processing is that of the 2015 code apart from the apostrophes in the contraction table) the code always normalized the prediction fully, but normalized the human answers only by the punctuation step, and only when they disagreed; it did not strip them first. Commit a013f00 made the processing of both sides the same and conditional. The 2021 challenge page links that commit and says the new evaluation server “might give slightly different accuracy results (difference of ~0.1%)” from the old one. In our reading the effect is small because the released answers are normalized already (above); models that output capitalized or punctuated answers are affected far more, as the example below shows.

Code and paper. The paper says all responses are lowercased and stripped of punctuation and articles before comparison, without conditions. The evaluation page lists the steps and attaches “This processing step is done for ground truth answers as well” to the punctuation step only, which matches the code before 2021; neither source mentions that processing is skipped for unanimous questions (in the current code). The page’s period rule (“except if it occurs as decimal”) is implemented as “unless a digit follows”, and its comma rule (no space between digits) as deleting every punctuation mark in an answer that contains a digit–comma–digit.

Example: consensus accuracy on rendered shapes

The script renders 300 images of 48 × 48 pixels per seed, each with one to five red, green, blue or yellow circles and squares, and asks three questions about each: “How many shapes are there?”, “What color is the biggest shape?” and “Is there a red shape?”. Ten simulated workers answer each question from the renderer’s ground truth: counts are right 80% of the time and otherwise off by one, colors are the plain name 80% of the time and otherwise a nearby name (“dark red”, “orange”), and yes/no answers are right 95% of the time. The answers are lowercase, like the released annotations.

Three predictors are scored. The pixel reader answers from the image: it labels connected foreground regions with SciPy, so touching shapes merge and it undercounts, and it reads colors from the pixels. The chatty predictor gives the same answers phrased as a generative model might (“Two.”, “Red.”, “Yes.”). The question-only predictor answers each question type with its most common answer on a separately generated set. Each is scored with the official accuracy as the 2021 and the pre-2021 code compute it, with exact match against the majority answer, with exact match after normalizing the prediction, and per answer type (2021 code).

import re
from collections import Counter
import numpy as np
from scipy import ndimage

# ---------- Answer processing, following vqaEval.py at commit a013f00 ----------
PUNCT = [";", "/", "[", "]", '"', "{", "}", "(", ")", "=", "+", "\\", "_", "-",
         ">", "<", "@", "`", ",", "?", "!"]
PERIOD = re.compile(r"(?!<=\d)(\.)(?!\d)")   # as written: drops '.' unless a digit follows
COMMA_DIGITS = re.compile(r"(\d)(,)(\d)")
NUMBERS = {"none": "0", "zero": "0", "one": "1", "two": "2", "three": "3", "four": "4",
           "five": "5", "six": "6", "seven": "7", "eight": "8", "nine": "9", "ten": "10"}
ARTICLES = {"a", "an", "the"}
CONTRACTIONS = {"dont": "don't", "isnt": "isn't", "arent": "aren't", "cant": "can't"}  # subset

def process_punctuation(s):
    out = s
    for p in PUNCT:   # the test looks at the input string, not at `out`
        if (p + " " in s or " " + p in s) or COMMA_DIGITS.search(s):
            out = out.replace(p, "")
        else:
            out = out.replace(p, " ")
    return PERIOD.sub("", out)

def process_digit_article(s):
    words = [NUMBERS.get(w, w) for w in s.lower().split()]
    return " ".join(CONTRACTIONS.get(w, w) for w in words if w not in ARTICLES)

def normalize(s):
    return process_digit_article(process_punctuation(s))

def vqa_accuracy(pred, humans, code="2021"):
    """Accuracy of one answer against ten human answers, as vqaEval.py computes it.
    code="2021": commit a013f00 (current); code="2016": commit 458a8b2 (before the fix)."""
    clean = lambda s: s.replace("\n", " ").replace("\t", " ").strip()
    pred = clean(pred)
    if code == "2021":
        humans = [clean(h) for h in humans]
        if len(set(humans)) > 1:          # normalize only when the humans disagree
            humans = [normalize(h) for h in humans]
            pred = normalize(pred)
    else:
        pred = normalize(pred)            # prediction always fully normalized
        if len(set(humans)) > 1:          # humans: punctuation step only
            humans = [process_punctuation(h) for h in humans]
    match = np.array([h == pred for h in humans])
    # leave one annotator out, score min(#matches among the other nine / 3, 1), average
    return float(np.mean([min(1.0, (match.sum() - m) / 3) for m in match]))

# ---------- Synthetic images, questions and ten annotators ----------
PALETTE = {"red": (0.9, 0.1, 0.1), "green": (0.1, 0.8, 0.1),
           "blue": (0.1, 0.2, 0.9), "yellow": (0.9, 0.9, 0.1)}
ALT = {"red": ["dark red", "orange"], "green": ["lime", "dark green"],
       "blue": ["navy", "purple"], "yellow": ["gold", "orange"]}
WORDS = ["zero", "one", "two", "three", "four", "five", "six"]
yy, xx = np.mgrid[0:48, 0:48]

def render(rng):
    """48x48 RGB image with 1-5 circles or squares of random color, size and position."""
    img = np.full((48, 48, 3), 0.5) + 0.03 * rng.standard_normal((48, 48, 3))
    shapes = []
    for _ in range(rng.integers(1, 6)):
        color = rng.choice(list(PALETTE))
        cy, cx, r = rng.uniform(6, 42), rng.uniform(6, 42), rng.uniform(3, 6)
        mask = ((yy - cy) ** 2 + (xx - cx) ** 2 < r ** 2) if rng.random() < 0.5 else \
               ((abs(yy - cy) < r) & (abs(xx - cx) < r))
        img[mask] = PALETTE[color]
        shapes.append((color, mask.sum()))
    return img, shapes

def make_questions(img, shapes, rng):
    """Three questions per image with ground truth from the renderer and ten answers each."""
    n = len(shapes)
    biggest = max(shapes, key=lambda s: s[1])[0]   # (drawn on top of overlaps or not)
    has_red = any(c == "red" for c, _ in shapes)
    qs = []
    # counting: annotators are right 80% of the time, otherwise off by one
    counts = [str(n if rng.random() < 0.8 else max(0, n + rng.choice([-1, 1]))) for _ in range(10)]
    qs.append(("how many", "number", "How many shapes are there?", counts))
    # color: the plain color name 80% of the time, otherwise a nearby name
    colors = [biggest if rng.random() < 0.8 else rng.choice(ALT[biggest]) for _ in range(10)]
    qs.append(("what color is the", "other", "What color is the biggest shape?", colors))
    # yes/no: right 95% of the time
    truth = "yes" if has_red else "no"
    yn = [truth if rng.random() < 0.95 else {"yes": "no", "no": "yes"}[truth] for _ in range(10)]
    qs.append(("is there a", "yes/no", "Is there a red shape?", yn))
    return qs

def read_image(img, qtype):
    """A 'model' that answers from the pixels: connected components and their colors.
    Touching shapes merge, so it undercounts and can misjudge the biggest shape."""
    fg = np.abs(img - 0.5).max(axis=2) > 0.2
    labels, n = ndimage.label(fg)
    if qtype == "how many":
        return str(n)
    names, rgb = list(PALETTE), np.array(list(PALETTE.values()))
    if qtype == "what color is the":
        sizes = ndimage.sum(fg, labels, range(1, n + 1))
        mean = img[labels == 1 + int(np.argmax(sizes))].mean(axis=0)
        return names[int(np.argmin(((rgb - mean) ** 2).sum(1)))]
    red = (np.abs(img - PALETTE["red"]).max(axis=2) < 0.15).sum()
    return "yes" if red > 10 else "no"

def chatty(ans):
    """The same answer as a generative model might phrase it."""
    return (WORDS[int(ans)] if ans.isdigit() else ans).capitalize() + "."

def run(seed, n_images=300):
    rng = np.random.default_rng(seed)
    data = []
    for _ in range(n_images):
        img, shapes = render(rng)
        for qtype, atype, q, humans in make_questions(img, shapes, rng):
            data.append((qtype, atype, q, humans, read_image(img, qtype)))
    # question-only baseline: most common answer per question type on a separate set
    prior_rng, prior = np.random.default_rng(seed + 1000), {}
    for _ in range(n_images):
        img, shapes = render(prior_rng)
        for qtype, _, _, humans in make_questions(img, shapes, prior_rng):
            prior.setdefault(qtype, Counter()).update(humans)
    prior = {t: c.most_common(1)[0][0] for t, c in prior.items()}
    predictors = {"pixel reader": lambda d: d[4],
                  "pixel reader, chatty": lambda d: chatty(d[4]),
                  "question only": lambda d: prior[d[0]]}
    rows = {}
    for name, f in predictors.items():
        preds = [f(d) for d in data]
        majority = [Counter(d[3]).most_common(1)[0][0] for d in data]
        rows[name] = {
            "VQA acc (2021 code)": np.mean([vqa_accuracy(p, d[3]) for p, d in zip(preds, data)]),
            "VQA acc (2016 code)": np.mean([vqa_accuracy(p, d[3], "2016") for p, d in zip(preds, data)]),
            "exact = majority": np.mean([p == m for p, m in zip(preds, majority)]),
            "normalized = majority": np.mean([normalize(p) == m for p, m in zip(preds, majority)]),
        }
        for atype in ("yes/no", "number", "other"):
            rows[name][atype] = np.mean([vqa_accuracy(p, d[3]) for p, d in zip(preds, data)
                                         if d[1] == atype])
    return rows, data

if __name__ == "__main__":
    print("matching humans (of 10) -> VQA accuracy:",
          {k: round(vqa_accuracy("2", ["2"] * k + ["3"] * (10 - k)), 3) for k in range(6)})
    print("'Yes' vs ten 'yes':", vqa_accuracy("Yes", ["yes"] * 10),
          "(2021 code),", vqa_accuracy("Yes", ["yes"] * 10, "2016"), "(2016 code)")
    print("'Yes' vs nine 'yes' and one 'no':", vqa_accuracy("Yes", ["yes"] * 9 + ["no"]))
    seeds = range(5)
    runs = [run(s)[0] for s in seeds]
    cols = list(runs[0]["pixel reader"])
    print(f"\n900 questions per seed; mean (min-max) over seeds {list(seeds)}")
    for name in runs[0]:
        print(name)
        for c in cols:
            v = np.array([r[name][c] for r in runs])
            print(f"   {c:22s} {v.mean():.3f} ({v.min():.3f}-{v.max():.3f})")

matching humans (of 10) -> VQA accuracy: {0: 0.0, 1: 0.3, 2: 0.6, 3: 0.9, 4: 1.0, 5: 1.0}
'Yes' vs ten 'yes': 0.0 (2021 code), 1.0 (2016 code)
'Yes' vs nine 'yes' and one 'no': 1.0

900 questions per seed; mean (min-max) over seeds [0, 1, 2, 3, 4]
pixel reader
   VQA acc (2021 code)    0.827 (0.813-0.852)
   VQA acc (2016 code)    0.827 (0.813-0.852)
   exact = majority       0.796 (0.781-0.820)
   normalized = majority  0.796 (0.781-0.820)
   yes/no                 0.999 (0.997-1.000)
   number                 0.622 (0.582-0.690)
   other                  0.859 (0.837-0.889)
pixel reader, chatty
   VQA acc (2021 code)    0.578 (0.557-0.600)
   VQA acc (2016 code)    0.827 (0.813-0.852)
   exact = majority       0.000 (0.000-0.000)
   normalized = majority  0.796 (0.781-0.820)
   yes/no                 0.392 (0.357-0.424)
   number                 0.561 (0.528-0.637)
   other                  0.781 (0.760-0.809)
question only
   VQA acc (2021 code)    0.391 (0.377-0.408)
   VQA acc (2016 code)    0.391 (0.377-0.408)
   exact = majority       0.331 (0.320-0.347)
   normalized = majority  0.331 (0.320-0.347)
   yes/no                 0.596 (0.563-0.614)
   number                 0.314 (0.303-0.329)
   other                  0.262 (0.240-0.287)

The consensus metric rates both lowercase predictors about 3 to 6 points higher than exact match against the majority answer. Partial credit outweighs the 0.9 cap at three votes: a miscount that two or three workers also made earns 0.6 or 0.9 instead of nothing. For the pixel reader, number questions are the weak spot, as merged shapes make it undercount, while the red-pixel test almost never fails.

The chatty predictor gives the same answers, yet the current code scores it about 25 points lower than the pre-2021 code over the five seeds. On questions where all ten workers agreed, mostly the yes/no ones, nothing is normalized, so “Yes.” does not match “yes”; where they disagreed, normalization turns “Two.” into “2” and “Red.” into “red”. Exact match against the majority answer gives it nothing, and normalizing the prediction first restores its full score. It follows that lowercasing and normalizing predictions before submission makes a model’s score independent of this behaviour of the code. The question-only predictor, at about 0.39, shows how much the question type alone gives on this toy data.

Cross-check against the official code. The same synthetic questions were written in the official JSON formats (annotations with ten answers each, questions, and a results list) and scored with the repository’s own vqa.py and vqaEval.py at commits a013f00 and 458a8b2, ported mechanically to Python 3 (print statements and tab expansion only). For the five seeds and both lowercase and chatty predictions, all 18,000 per-question scores and every overall and per-answer-type score agreed with the reimplementation to the code’s two decimals. So did 2,000 further questions built from deliberately messy strings with capitals, punctuation, number words, articles, contractions, decimals and thousands separators on both sides. The contraction table in the shown code is a subset of the official one; the messy strings used only contractions it contains.

Evaluation servers and submission rules

Test answers are scored only on the evaluation servers. Results are a JSON list of {"question_id": int, "answer": str}, one per test question (evaluation page).

• VQA v1 (2016 challenge, CodaLab). Four competitions (open-ended and multiple choice, real and abstract). Test-dev allowed at most 10 submissions per day; the full test set was limited to one upload per day and five per user, with multiple accounts for one project not allowed. Papers were asked to report test-standard accuracy, obtained from either of the two non-test-dev phases, and to compare with the test2015 leaderboards (2016 challenge page). The extended arXiv version of the paper (v7, 2016) instead describes test-dev as allowing unlimited submissions.
• VQA v2 (2017–2021 challenges, EvalAI). Phases Test-Dev, Test-Standard and Test-Challenge (in 2017 named test-dev2017, test2017 and Challenge test2017). Test-Dev allows 10 submissions per day; Test-Standard and Test-Challenge one per day and five in total, with the limits applying per team. A Test-Standard or Test-Challenge submission returns accuracies on both test-standard and test-dev; only Test-Standard has a public leaderboard, and a public Test-Standard entry made before a deadline counted as a challenge entry (2017 and 2021 challenge pages).
• As of October 2026, EvalAI’s public listing for the 2021 challenge shows the Test-Dev and Test-Standard phases open, with end dates in 2099 and the limits above, and the Test-Challenge phase closed on 7 May 2021. The site lists no challenge after 2021, the sixth.

Test-reserve exists to detect overfitting: a large gap between a method’s test-standard and test-reserve scores “will raise a red-flag”, and test-reserve results are never revealed (2021 challenge page). Benchmarking describes this design, which VQA took from COCO, and benchmark overfitting the problem it guards against.

License

As of October 2026, the VQA terms page states that the annotations “belong to the VQA Consortium” and are licensed under a Creative Commons Attribution 4.0 International License. The software is under a BSD-style licence (“Copyright © 2015, VQA Consortium”); the repository’s own license.txt is a BSD-style licence with “Copyright (c) 2014, Aishwarya Agrawal”. The download page says that downloading the dataset means agreeing to these terms.

The terms page says nothing about images. The real images are COCO’s, downloaded from the COCO site: according to the COCO terms of use (as of October 2026), the COCO Consortium does not own their copyright, their use must abide by the Flickr Terms of Use, and users take full responsibility for it. For the abstract scenes, neither the VQA terms nor the abstract-scenes repository states a licence for the rendered images, the clipart or the composition files, so their terms are unclear.

Limitations and Bias

• Language priors. In v1, answering “yes” to everything scored 29.72%, the most common answer per question type 36.18%, and people shown only the question 40.81% on real images; an LSTM on the question alone reached 48.76% on test-dev (Antol et al., 2015). Goyal et al. (2017) found that “tennis” answers 41% of “What sport is” questions and “2” 39% of “How many” questions, that “is there a clock” questions are answered “yes” 98% of the time, and that answering “yes” to questions starting “Do you see a” gives 87% accuracy. They also cite a visual priming bias identified by Zhang et al. (2016): workers wrote questions while looking at the image, so a question like “Is there a clock tower in the picture?” is asked only about images that contain one.
• Balancing is partial. In v2, 22% of questions found no complementary image, the majority answer did not change for about 9% of questions, and the authors describe the result as alleviating language bias, “though not elimination” (Goyal et al., 2017). Language-only accuracy fell from 48.21% to 41.40% when a model trained on v1 was tested on balanced validation data.
• Exact matching. Agreement among people was 83.30% on real images and 87.49% on abstract scenes (test-standard), above 95% on yes/no questions and below 76% on others, partly because synonyms and plurals do not match (Antol et al., 2015). Correct paraphrases and full sentences earn nothing, and the normalization depends on the code version (Evaluation).
• Hidden test answers. Analyses that need answers must re-split train and validation; VQA-CP did so for that reason (Agrawal et al., 2018).
• Scope. Questions and answers are in English. The real images are COCO’s Flickr photographs, with COCO’s content and biases; see COCO and dataset bias. The VQA papers and pages read for this article (October 2026) say nothing about the consent of people pictured, about demographics, or about removal requests.
• Long reuse of one test set. Test-standard has been the comparison point since 2017 for v2, and the servers remain open.

Common Uses

• Benchmarking VQA models, reporting open-ended accuracy on v2 test-dev or test-standard, usually with the yes/no, number and other breakdown.
• A fine-tuning task for vision–language pretraining, as that article describes, and a training source in the instruction mixtures of vision–language models; LLaVA-1.5, for example, includes VQAv2 in its training mix, so its VQAv2 result is reported with training images seen.
• Studying language priors and dataset bias: the complementary pairs allow pairwise analyses (Goyal et al. report how often a model answers both images of a pair correctly), and VQA-CP re-splits v1 and v2 to change the answer priors between training and test.

Historical Milestones

• 2015: the dataset and its baselines. Antol et al.’s best model, an LSTM question encoder fused with VGGNet image features, reached 54.06% on v1 real-image test-standard (open-ended), against 83.30% for people. The extended arXiv version reports 58.16% for an improved “deeper LSTM Q + norm I” model, first in v5 (March 2016) (see the paper article for the versions).
• 2016: the first challenge. Results were announced at a CVPR 2016 workshop; multimodal compact bilinear pooling (MCB) won the real-image track (Goyal et al., 2017; Agrawal et al., 2018). Fukui et al. report 66.5% open-ended and 70.1% multiple choice on test-standard for an ensemble of seven MCB models with attention, trained on train and val, some also with Visual Genome data.
• 2017: VQA v2. Trained on v2 train and val, the language-only baseline scored 44.26%, the v1 baseline model 54.22% and MCB 62.27% on v2 test-standard (Goyal et al., arXiv v3, Table 2; the CVPR version reports validation results only). The 2017 challenge was won by the bottom-up and top-down attention model, an ensemble of 30 models scoring 70.34% on test-standard as of 8 August 2017 (Anderson et al., 2018).
• 2018: changing priors. VQA-CP re-split v1 and v2 so that each question type’s answer distribution differs between training and test (Agrawal et al., 2018); the question-only model scored 43.01% on VQA v2 val when trained on v2 train, but 15.95% on VQA-CP v2 test when trained on VQA-CP v2 train, and MCB 59.71% and 36.33%.
• 2019 onward: pretrained models. Pretrained transformers and then vision–language models pushed v2 test-standard accuracy into the 70s and 80s; Visual Question Answering tabulates them with their settings.
• 2021: the last challenge and the evaluation fix. The sixth challenge ran on EvalAI with the corrected evaluation code.
• 2025: the Mark Everingham Prize, awarded at ICCV 2025 to the VQA challenge series (A. Agrawal, Y. Goyal, A. Shrivastava, D. Batra, D. Parikh and collaborators).

Related Datasets

• VQA-CP v1 and v2 (Agrawal et al., 2018) regroup the questions of VQA train and val by question type and answer and assign whole groups to training or test, so test answers per question type differ from training ones; VQA-CP v2 train has about 438K questions and test about 220K.
• Balanced binary abstract scenes (Zhang et al., 2016) pair each yes/no question with two clipart scenes that differ in the answer, made by asking workers to change the scene as little as possible so that the answer flips; they are distributed on the VQA site.
• Visual Genome contains question–answer pairs about its images and has been used as extra VQA training data: the Up-Down model added about 485K Visual Genome questions whose answers were in its vocabulary (Anderson et al., 2018).
• Later VQA benchmarks target compositional reasoning (CLEVR, GQA), outside knowledge (OK-VQA, on COCO images), text in images (TextVQA) and questions from blind users (VizWiz); Visual Question Answering compares them.
• COCO supplies the real images and their five captions per image.

Resources

• Official site, with release history, downloads, evaluation page, terms and challenge pages: visualqa.org.
• Python API and evaluation code: GT-Vision-Lab/VQA, current evaluation at commit a013f00 (Python 2.7).
• Evaluation server and leaderboards: VQA Challenge 2021 on EvalAI; earlier test-standard leaderboards are linked from the 2021 challenge page.
• Abstract scene rendering code: GT-Vision-Lab/abstract_scenes_v002.