Visual Question Answering

Visual question answering (VQA) is the task of answering a natural-language question about an image: given a photograph of a street and the question “what color is the bus?”, output “red”. Antol et al. (2015) proposed it as a test of image understanding that, unlike captioning, can be scored automatically, since one to three words answer many questions. Within multimodal learning, it is the standard example of a task that needs multimodal fusion: neither input alone determines the answer.

Problem Definition

Given an image $I$ and a question $q$, predict an answer $a$. Early models, and many fine-tuned pretrained transformers, treat the task as classification over a fixed vocabulary $\mathcal{A}$ of frequent training answers,

$$\hat{a} = \arg\max_{a \in \mathcal{A}} \; p_\theta(a \mid I, q),$$

so an answer outside $\mathcal{A}$ can never be produced. Antol et al.’s baselines used the 1,000 most frequent answers, which cover 82.67% of the training and validation answers; Anderson et al. (2018) kept correct training answers that appear more than 8 times, 3,129 answers on VQA v2. Generative models instead produce the answer token by token, $p_\theta(a \mid I, q) = \prod_t p_\theta(a_t \mid a_{<t}, I, q)$, which removes the vocabulary limit but makes scoring harder.

Besides this open-ended setting, the original benchmark had a multiple-choice one with 18 candidate answers per question (Antol et al., 2015).

Input and Output

Questions are binary (“is the umbrella upside down?”), counting (“how many children are in the bed?”) or open (“who is wearing glasses?”), and results are reported separately for yes/no, number and other answers. Answers are short: in VQA, 89.32% of the answers for real images are one word (Antol et al., 2015). The image enters as a grid of CNN features, detected-region features from a Faster R-CNN, or Vision Transformer patch tokens.

Variants

Specialized benchmarks (below) target compositional reasoning over several steps (CLEVR, GQA), outside knowledge (OK-VQA), text in the image (TextVQA, and DocVQA and ChartQA for documents and charts), and photos taken by blind users (VizWiz).

Challenges

• Language priors: answers are predictable from the question alone, so models can score well without looking (below).
• Answer ambiguity: people disagree on synonyms (“maroon”, “dark red”) and counts, hence the ten human answers per question.
• Skills beyond recognition: counting, reading text, fine detail and outside knowledge.

Approaches

Joint embedding baselines

Antol et al. (2015) encoded the image with VGGNet’s 4,096-dimensional last hidden layer and the question with an LSTM, fused them by element-wise multiplication and applied a softmax over the top 1,000 answers. This LSTM Q+I model reached 54.06% on VQA test-standard, but an LSTM on the question alone already reached 48.76% (test-dev).

Attention over the image

A single image vector must summarize everything a question might ask. Stacked attention networks (SAN; Yang et al., 2016) kept the $14 \times 14$ grid of 512-dimensional VGG features of a $448 \times 448$ image and let the question vector $u^0 = v_Q$ attend over the 196 regions $v_i$ several times,

$$p^k = \mathrm{softmax}\Big(W_P^k \tanh\big(W_{I}^k V \oplus (W_{Q}^k u^{k-1} + b^k)\big) + b_P^k\Big), \qquad u^k = \sum_i p^k_i v_i + u^{k-1},$$

where $\oplus$ adds the vector to every column of the matrix. With two attention layers, SAN reached 58.7% on VQA test-dev and 58.9% on test-standard, and the second layer’s attention was sharper.

Bottom-up and top-down attention (Up-Down; Anderson et al., 2018) attends over object regions from a Faster R-CNN trained on Visual Genome, the bottom-up features described in image captioning and vision–language pretraining. On the VQA v2 validation set a single Up-Down model scored 63.2%, against 59.4% for the best ResNet grid baseline; an ensemble of 30 models reached 70.34% on test-standard and took first place in the 2017 VQA Challenge. Its companion paper (Teney et al., 2018) listed the details that mattered, including independent sigmoid outputs instead of a softmax, so several answers can be correct, trained with binary cross-entropy against soft scores $s_{ij} \in [0, 1]$ that reflect how many annotators gave answer $j$ to question $i$:

$$\mathcal{L} = -\sum_{i} \sum_{j \in \mathcal{A}} \Big[ s_{ij} \log \hat{s}_{ij} + (1 - s_{ij}) \log (1 - \hat{s}_{ij}) \Big].$$

About 7% of their training questions had no correct answer in the 3,129-answer vocabulary. ViLBERT kept this formulation.

Fusion operators

Many VQA models of 2016–2017 differed mainly in how image and question vectors are combined: compact and low-rank bilinear pooling, co-attention, feature-wise modulation. Multimodal Fusion compares these operators with their VQA numbers.

Modular and compositional models

Neural module networks (NMN; Andreas et al., 2016) build a different network for each question. A dependency parse of “what color is his tie?” becomes the layout describe[color](find[tie]), assembled from jointly trained modules: find produces an attention map, transform and combine manipulate attention maps, and describe and measure map them to answers. With an LSTM question encoder added and fine-tuned image features, NMN reached 58.6% on VQA test-dev. Multi-step reasoning of this kind is what the synthetic CLEVR benchmark (below) diagnoses; the FiLM model in the fusion article was evaluated there.

Pretrained transformers

From 2019, VQA became a fine-tuning task for vision–language pretraining. ViLBERT (Lu et al., 2019) fine-tuned a two-layer MLP over 3,129 answers with soft scores; LXMERT (Tan & Bansal, 2019) also pretrained on question answering data, and UNITER (Chen et al., 2020) used a single stream. ALBEF and BLIP replaced the classifier with an answer decoder, but at inference both restricted it to 3,128 candidate answers (Li et al., 2021, NeurIPS version, whose arXiv version says 3,192; Li et al., 2022), so their outputs still come from a fixed list. A CLIP-style dual encoder alone does not fit the task: image and question never interact.

Vision–language models

Vision–language models connect a pretrained image encoder to a large language model and answer in free text. Flamingo (Alayrac et al., 2022) answered open-ended questions with beam search after a prompt of the form “Question: … Answer: …”, preceded by a few solved examples with images (few-shot) or, for its zero-shot results, by two text-only examples with the images removed. BLIP-2 (Li et al., 2023) evaluated zero-shot with the prompt “Question: {} Short answer:” for its FlanT5 models and a length penalty that favors short answers, which the authors say align better with human annotation.

Results on VQA v2

Model	Year	Setting	test-dev	test-std
d-LSTM Q + norm I (as listed by Anderson et al.)	—	trained on VQA v2	—	54.22
MCB (as listed by Anderson et al.)	—	trained on VQA v2	—	62.27
Up-Down, ensemble of 30	2017	trained on VQA v2 + Visual Genome	—	70.34
ViLBERT	2019	pretrained, fine-tuned	70.55	70.92
LXMERT	2019	pretrained, fine-tuned	72.4	72.5
UNITER-large	2020	pretrained, fine-tuned	73.82	74.02
BLIP (ViT-B, 129M images, CapFilt-L)	2022	pretrained, fine-tuned	78.25	78.32
Flamingo (80B)	2022	zero-shot / 4-shot / 32-shot	56.3 / 63.1 / 67.6	—
Flamingo (80B)	2022	fine-tuned	82.0	82.1
BLIP-2 (ViT-g, FlanT5-XXL)	2023	zero-shot	65.0	—
BLIP-2 (ViT-g, OPT-6.7B)	2023	fine-tuned	82.19	82.30

VQA accuracy in percent as printed; the first two rows are baselines listed in Anderson et al.’s table (as at 8 August 2017), which credits them to Goyal et al., whose arXiv version reports them trained on VQA v2 train+val. BLIP-2’s fine-tuned columns are headed test-dev / test-std in its arXiv version but val / test-dev in the ICML version. The rows differ in training data, features and model size, and Flamingo’s authors used VQA v2 for design decisions, so their estimates there may be optimistic. For scale, people answering VQA v1 questions about real images reached 83.30% (Antol et al., 2015).

Language Priors

In VQA v1, answering “yes” to every question scored 29.72%, and the most common answer for each question type 36.18%; even people shown only the question reached 40.81% (Antol et al., 2015). Goyal et al. (2017) found that questions starting with “is there a clock” were answered “yes” 98% of the time.

Balanced pairs. VQA v2 (Goyal et al., 2017) counters this by pairing each question with a complementary image: annotators chose, among similar COCO images, one for which the same question has a different answer. Models trained on the original data lost accuracy on balanced validation data: a language-only model fell from 48.21% to 41.40% and MCB from 60.36% to 54.22%.

Changing priors. VQA-CP (Agrawal et al., 2018) re-splits the VQA v1 and v2 training and validation data so that groups of questions sharing a type and an answer, such as “what color is the…” with “white”, go entirely to the training or the test side; each question type’s answer distribution then differs between the two. On VQA-CP v2, the question-only d-LSTM Q fell from 43.01% (VQA v2 val) to 15.95%, SAN from 52.02% to 24.96% and MCB from 59.71% to 36.33%. The authors’ GVQA, which separates recognizing visual concepts from identifying the plausible answers to a question, reached 31.30%.

Example: a question-only baseline under shifting priors

The script makes toy VQA data: $16 \times 16$ RGB images of one red, green, blue or yellow circle, square or triangle at a random position, size and brightness, with heavy pixel noise, two invented questions (“what color is the shape?”, “what shape is it?”) and seven answers. In training, 70% of color questions are answered “red” and 70% of shape questions “circle”. The test splits keep this prior, make it uniform (balanced), or exclude the training majority answers (changed prior, as in VQA-CP).

The question-only model predicts each question type’s most frequent training answer, the best an image-blind predictor can do on the training distribution. The image + question model feeds the $2 \times 2$-pooled pixels (192 values) concatenated with the question’s one-hot code to a one-hidden-layer network, trained with full-batch Adam on cross-entropy; its hand-written gradients are checked against finite differences. Each seed trains on 2,000 examples; the script runs in about 10 seconds with NumPy 2.3 and printed identical output on two runs.

import os
import sys
os.environ["OMP_NUM_THREADS"] = os.environ["OPENBLAS_NUM_THREADS"] = os.environ["MKL_NUM_THREADS"] = "1"
import numpy as np                                # one BLAS thread: faster here, same output

S = 16
COLORS = {"red": (1, .2, .2), "green": (.2, 1, .2), "blue": (.2, .2, 1), "yellow": (1, 1, .2)}
SHAPES = ("circle", "square", "triangle")
yy, xx = np.mgrid[0:S, 0:S] + 0.5

# P(answer | question type) in each split. Training is biased: "what color?" is
# usually "red" and "what shape?" usually "circle".
PRIORS = {
    "train":          ([.70, .10, .10, .10], [.70, .15, .15]),
    "same prior":     ([.70, .10, .10, .10], [.70, .15, .15]),
    "balanced":       ([.25, .25, .25, .25], [1/3, 1/3, 1/3]),
    "changed prior":  ([0, 1/3, 1/3, 1/3],   [0, .5, .5]),   # VQA-CP-like: majority never appears
}

def make_split(n, split, rng):
    """n (image, question, answer) triples: one colored shape per 16x16 RGB image."""
    pc, ps = PRIORS[split]
    qtype = rng.integers(2, size=n)                       # 0: "what color is the shape?", 1: "what shape is it?"
    color = np.where(qtype == 0, rng.choice(4, n, p=pc), rng.integers(4, size=n))
    shape = np.where(qtype == 1, rng.choice(3, n, p=ps), rng.integers(3, size=n))
    cy, cx = 8 + rng.uniform(-2, 2, size=(2, n, 1, 1))
    r = rng.uniform(3.5, 5.5, size=(n, 1, 1))
    y, x = yy - cy, xx - cx
    masks = np.stack([x**2 + y**2 < r**2,
                      (abs(x) < 0.85 * r) & (abs(y) < 0.85 * r),
                      (y < 0.8 * r) & (y > -r + 2 * abs(x))])
    m = masks[shape, np.arange(n)]
    tint = np.array(list(COLORS.values()))[color] * rng.uniform(0.5, 1.0, size=(n, 1))
    img = 0.1 + m[..., None] * tint[:, None, None, :] + 0.2 * rng.standard_normal((n, S, S, 3))
    img = img.reshape(n, 8, 2, 8, 2, 3).mean((2, 4)).reshape(n, -1)      # 2x2 average pool -> 192 values
    answer = np.where(qtype == 0, color, 4 + shape)     # 7 answers: 4 colors, then 3 shapes
    return img, np.eye(2)[qtype], answer

def softmax(z):
    z = np.exp(z - z.max(1, keepdims=True))
    return z / z.sum(1, keepdims=True)

def forward(P, X, Q):
    """Image + question model: concatenate pooled pixels and question one-hot, one hidden layer."""
    H = np.maximum(0, np.hstack([X, Q]) @ P["W1"] + P["b1"])
    return H, H @ P["W2"] + P["b2"]

def loss_grad(P, X, Q, y):
    H, Z = forward(P, X, Q)
    Pr = softmax(Z)
    loss = -np.log(Pr[np.arange(len(y)), y]).mean()
    dZ = Pr.copy()
    dZ[np.arange(len(y)), y] -= 1
    dZ /= len(y)
    dH = (dZ @ P["W2"].T) * (H > 0)
    XQ = np.hstack([X, Q])
    return loss, {"W1": XQ.T @ dH, "b1": dH.sum(0), "W2": H.T @ dZ, "b2": dZ.sum(0)}

def train(X, Q, y, rng, hidden=64, steps=600, lr=3e-3):
    d = X.shape[1] + 2
    P = {"W1": rng.standard_normal((d, hidden)) * np.sqrt(2 / d), "b1": np.zeros(hidden),
         "W2": rng.standard_normal((hidden, 7)) * 0.1, "b2": np.zeros(7)}
    m = {k: np.zeros_like(v) for k, v in P.items()}
    v = {k: np.zeros_like(v) for k, v in P.items()}
    for t in range(1, steps + 1):                        # full-batch Adam
        _, g = loss_grad(P, X, Q, y)
        for k in P:
            m[k] = 0.9 * m[k] + 0.1 * g[k]
            v[k] = 0.999 * v[k] + 0.001 * g[k] ** 2
            P[k] -= lr * (m[k] / (1 - 0.9**t)) / (np.sqrt(v[k] / (1 - 0.999**t)) + 1e-8)
    return P

def grad_check(rng):
    X, Q, y = make_split(20, "train", rng)
    P = train(X, Q, y, rng, steps=1)
    _, g = loss_grad(P, X, Q, y)
    err = 0.0
    for k in P:
        for _ in range(5):
            i = tuple(rng.integers(0, s) for s in P[k].shape)
            old = P[k][i]
            P[k][i] = old + 1e-6; lp = loss_grad(P, X, Q, y)[0]
            P[k][i] = old - 1e-6; lm = loss_grad(P, X, Q, y)[0]
            P[k][i] = old
            err = max(err, abs((lp - lm) / 2e-6 - g[k][i]))
    return err

def run(seed, n_train=2000, n_test=2000):
    rng = np.random.default_rng(seed)
    X, Q, y = make_split(n_train, "train", rng)
    # Question-only model: the most frequent training answer for each question type
    # (the best any predictor that ignores the image can do on the training distribution).
    q_only = [np.bincount(y[Q[:, t] == 1], minlength=7).argmax() for t in range(2)]
    P = train(X, Q, y, rng)
    out = {}
    for split in ("same prior", "balanced", "changed prior"):
        Xt, Qt, yt = make_split(n_test, split, np.random.default_rng(1000 + seed))
        pred_q = np.array(q_only)[Qt.argmax(1)]
        pred_iq = forward(P, Xt, Qt)[1].argmax(1)
        out[split] = ((pred_q == yt).mean(), (pred_iq == yt).mean())
        if split == "changed prior":          # here "red" and "circle" are never correct
            majority = np.isin(pred_iq, q_only).mean()
    return out, majority

if __name__ == "__main__":
    seeds = [int(a) for a in sys.argv[1:]] or [0, 1, 2, 3, 4]
    print(f"max |analytic - finite-difference gradient|: {grad_check(np.random.default_rng(99)):.1e}")
    runs = [run(s) for s in seeds]
    results = [r for r, _ in runs]
    print(f"accuracy on 2,000 test questions, mean (min-max) over seeds {seeds}")
    print(f"{'test split':14s} {'question only':>22s} {'image + question':>22s}")
    for split in results[0]:
        a = np.array([r[split] for r in results])
        cell = lambda c: f"{a[:, c].mean():.3f} ({a[:, c].min():.3f}-{a[:, c].max():.3f})"
        print(f"{split:14s} {cell(0):>22s} {cell(1):>22s}")
    print("changed prior: image + question model answers 'red' or 'circle' on "
          + ", ".join(f"{m:.1%}" for _, m in runs) + " of questions")

max |analytic - finite-difference gradient|: 2.4e-10
accuracy on 2,000 test questions, mean (min-max) over seeds [0, 1, 2, 3, 4]
test split              question only       image + question
same prior        0.701 (0.693-0.706)    0.880 (0.865-0.893)
balanced          0.293 (0.287-0.307)    0.819 (0.801-0.837)
changed prior     0.000 (0.000-0.000)    0.755 (0.738-0.769)
changed prior: image + question model answers 'red' or 'circle' on 17.8%, 20.0%, 19.5%, 19.3%, 21.6% of questions

With the training prior, the question-only model looks respectable at 70%. On the balanced split it falls to 29%, the chance level (the mean of 1/4 and 1/3), and on the changed-prior split to zero. The image + question model holds up far better (88%, 82%, 76%) but is not prior-free: on the changed-prior split 18–22% of its answers are “red” or “circle”, which are never correct there, so most of its errors are the training prior, presumably where the noisy image is ambiguous, the pattern VQA-CP measured in real models.

Datasets and Benchmarks

Dataset	Year	Images	Questions	Focus
VQA (v1)	2015	204,721 COCO + 50,000 abstract scenes	about 0.76M, 10 answers each	open-ended and multiple-choice
VQA v2	2017	about 200K COCO	about 1.1M, 10 answers each	complementary pairs against priors
CLEVR	2017	100,000 rendered	999,968 (853,554 unique)	synthetic compositional reasoning
VizWiz	2018	photos taken by blind users	31,173, 10 answers each	spoken questions; many unanswerable
VQA-CP v1, v2	2018	re-splits of VQA	same as source	changed answer priors
GQA	2019	113K (Visual Genome)	22M generated, 1.7M balanced	real-image compositional reasoning
OK-VQA	2019	14,031 COCO	14,055, 5 answers each	outside knowledge
TextVQA	2019	28,408 Open Images	45,336, 10 answers each	reading text in images
DocVQA	2021	12,000+ documents	50,000	document images
ChartQA	2022	charts	9.6K human-written + 23.1K generated	charts, arithmetic over data

• CLEVR (Johnson et al., 2017) renders simple 3D objects and generates each question from a functional program, so the reasoning steps are known. On attribute queries, question-only baselines scored near chance, showing little question-conditional bias.
• GQA (Hudson & Manning, 2019) generates questions with functional programs from Visual Genome scene graphs and balances answer distributions. The authors reported 42.1% for a blind LSTM in the abstract (41.07% in the test-set table), 54.1% for strong VQA models and 89.3% for people, and added metrics such as consistency and grounding.
• OK-VQA (Marino et al., 2019) limited answer bias by removing questions whose answer was too common; the best non-oracle baseline, MUTAN combined with ArticleNet, which retrieves Wikipedia articles, scored 27.84%. Its five answers per question are each counted twice.
• VizWiz (Gurari et al., 2018): crowd workers judged about 28% of the blind users’ questions unanswerable, for instance because of blur or framing. TextVQA (Singh et al., 2019) introduced LoRRA, which can copy a token read by an OCR system as its answer.

Evaluation Metrics

VQA accuracy. Unlike plain accuracy, the VQA metric gives partial credit. With ten human answers $h_1, \dots, h_{10}$ per question, an answer counts as fully correct if at least three annotators gave it (Antol et al., 2015). To be consistent with human accuracy, machine accuracy is averaged over the $\binom{10}{9} = 10$ subsets of nine annotators:

$$\mathrm{Acc}(a) = \frac{1}{10} \sum_{k=1}^{10} \min\left(1, \frac{\big|\{\, j \neq k : h_j = a \,\}\big|}{3}\right).$$

An answer given by 0, 1, 2, 3 or at least 4 annotators scores 0, 0.3, 0.6, 0.9 or 1. Antol et al. state that answers are lowercased, number words converted to digits, and punctuation and articles removed before comparison; the official vqaEval.py applies this only when the ten human answers are not all identical. Antol et al. rejected soft metrics based on word embeddings because they would merge answers such as “left” and “right”. The script implements the metric, with normalization following the official vqaEval.py, on invented annotator answers.

import re
import numpy as np

# Answer normalization, following the official VQA evaluation script (vqaEval.py).
NUMBERS = {"none": "0", "zero": "0", "one": "1", "two": "2", "three": "3", "four": "4",
           "five": "5", "six": "6", "seven": "7", "eight": "8", "nine": "9", "ten": "10"}
ARTICLES = {"a", "an", "the"}
CONTRACTIONS = {"dont": "don't", "doesnt": "doesn't", "isnt": "isn't", "cant": "can't"}  # subset
PUNCT = [";", "/", "[", "]", '"', "{", "}", "(", ")", "=", "+", "\\", "_", "-",
         ">", "<", "@", "`", ",", "?", "!"]
PERIOD = re.compile(r"(?!<=\d)(\.)(?!\d)")   # as written in vqaEval.py: drops '.' unless a digit follows
COMMA_IN_NUMBER = re.compile(r"(\d)(,)(\d)")

def normalize(ans):
    ans = ans.replace("\n", " ").replace("\t", " ").strip()
    out = ans
    for p in PUNCT:
        if (p + " " in ans or " " + p in ans) or COMMA_IN_NUMBER.search(ans):
            out = out.replace(p, "")
        else:
            out = out.replace(p, " ")
    out = PERIOD.sub("", out)
    words = [NUMBERS.get(w, w) for w in out.lower().split()]
    words = [CONTRACTIONS.get(w, w) for w in words if w not in ARTICLES]
    return " ".join(words)

def vqa_accuracy(pred, humans, official=True):
    """Accuracy of one predicted answer against 10 human answers: min(#matches/3, 1),
    averaged over the 10 leave-one-out subsets of 9 annotators. With official=True the
    answers are normalized only when the humans disagree, as in vqaEval.py."""
    if not official or len(set(humans)) > 1:
        humans, pred = [normalize(h) for h in humans], normalize(pred)
    match = np.array([h == pred for h in humans])
    loo = [min(1.0, (match.sum() - match[i]) / 3) for i in range(len(humans))]
    return float(np.mean(loo)), int(match.sum())

# Invented answers from 10 annotators for three questions about one street photo.
questions = {
    "How many dogs are on the sofa?":
        ["2", "2", "two", "2", "2 dogs", "2", "3", "2", "Two", "2"],
    "What color is the bus?":
        ["red", "red", "maroon", "red", "dark red", "maroon", "maroon", "Red.", "red and white", "burgundy"],
    "Is the traffic light green?": ["yes"] * 10,
}
predictions = {
    "How many dogs are on the sofa?": ["two", "3", "2 dogs"],
    "What color is the bus?": ["Red", "maroon", "dark red", "pink", "The bus is red."],
    "Is the traffic light green?": ["yes", "Yes."],
}

print(f"{'question':32s} {'prediction':17s} {'matches':>7s} {'min(n/3,1)':>10s} {'VQA acc':>8s}")
for q, preds in predictions.items():
    for p in preds:
        acc, n = vqa_accuracy(p, questions[q])
        print(f"{q:32s} {p!r:17s} {n:7d} {min(n / 3, 1):10.3f} {acc:8.3f}")

print("\nmatching humans (of 10) -> VQA accuracy:",
      {n: round(float(np.mean([min(1, (n - (i < n)) / 3) for i in range(10)])), 3) for n in range(6)})
print("'Yes.' vs ten 'yes', always normalizing:",
      vqa_accuracy("Yes.", questions["Is the traffic light green?"], official=False)[0])

question                         prediction        matches min(n/3,1)  VQA acc
How many dogs are on the sofa?   'two'                   8      1.000    1.000
How many dogs are on the sofa?   '3'                     1      0.333    0.300
How many dogs are on the sofa?   '2 dogs'                1      0.333    0.300
What color is the bus?           'Red'                   4      1.000    1.000
What color is the bus?           'maroon'                3      1.000    0.900
What color is the bus?           'dark red'              1      0.333    0.300
What color is the bus?           'pink'                  0      0.000    0.000
What color is the bus?           'The bus is red.'       0      0.000    0.000
Is the traffic light green?      'yes'                  10      1.000    1.000
Is the traffic light green?      'Yes.'                  0      0.000    0.000

matching humans (of 10) -> VQA accuracy: {0: 0.0, 1: 0.3, 2: 0.6, 3: 0.9, 4: 1.0, 5: 1.0}
'Yes.' vs ten 'yes', always normalizing: 1.0

Normalization lets “two” match “2” and counts “Red.” as “red”, and the leave-one-out average turns exactly three matches into 0.9 rather than 1. The metric is literal: “The bus is red.” scores zero, because normalization leaves “bus is red”, which no annotator wrote. And “Yes.” scores zero against ten “yes” answers: with unanimous annotators nothing is normalized, so the answer must match exactly; always normalizing would give 1.

Scoring generated answers. The metric rewards reproducing an annotator’s short phrasing, so generative models are steered toward that format: ALBEF and BLIP restrict their decoders to a candidate list, Flamingo’s few-shot prompts show short answers, and BLIP-2 asks for a “short answer” with a length penalty. In our reading, zero-shot numbers therefore measure answer format as well as understanding, and comparisons across prompting schemes need care.

Other metrics. DocVQA uses average normalized Levenshtein similarity (ANLS), which gives partial credit to answers with small OCR errors (Mathew et al., 2021). ChartQA counts a numeric answer as correct within 5% of the gold value and requires an exact match otherwise (Masry et al., 2022). VizWiz also evaluates whether a model can tell that a question is unanswerable.

Applications

• Assistance for blind and low-vision people, the setting VizWiz collects and a motivation named by Antol et al.
• Document and chart understanding: reading values from document images and plots by asking questions (DocVQA, ChartQA).
• Evaluating vision–language models: Flamingo and BLIP-2, for example, report VQA benchmarks beside captioning and retrieval.

Open Problems

• Grounding versus shortcuts. VQA v2, VQA-CP, CLEVR and GQA each remove some shortcuts; in our view none removes all.
• Free-form answers. Exact matching penalizes correct sentences, and looser matching risks rewarding wrong ones.
• Reading and counting remain weak, as the low number accuracies of early models and the human–machine gap reported on TextVQA show.
• Knowledge and hallucination. BLIP-2’s authors attribute Flamingo’s lead on OK-VQA to the knowledge of its larger language model; generated answers can also describe content absent from the image (multimodal hallucination).
• Unanswerable questions. VizWiz shows that real users often ask questions their photo cannot answer.