Visual Reasoning

Visual reasoning is the task of answering a question about an image, or judging whether a sentence is true of it, when the answer needs several steps of perception composed in the right order: find the objects a phrase describes, read their attributes, follow a spatial relation, count, compare, combine with and, or or not. “What color is the cube to the right of the yellow sphere?” (Johnson et al., 2017, CLEVR) cannot be answered by recognizing one object; it needs a small program. The task overlaps with visual question answering, which covers the general problem, the VQA benchmarks and their language priors, and it is often called compositional or relational visual reasoning. Its research has been driven by diagnostic benchmarks built so that shortcuts fail, and by a recurring finding: models that score well on a benchmark’s usual test set often fail on new combinations of familiar parts.

Problem Definition

Given an image $I$ (or a pair of images) and a query $q$, predict an output $y$ whose correct value is determined by a composition of elementary operations on the scene. CLEVR makes this explicit: every question comes with a functional program built from basic functions such as filter_color, relate, count, equal and query_shape, and executing the program on the image’s ground-truth scene graph yields the answer (Johnson et al., 2017, CLEVR). “How many red things are there?” is count(filter_color(red, scene())); programs may be chains or trees joined by logical operators.

Writing $z = \pi(q)$ for the program a question denotes and $s = \sigma(I)$ for the scene, the answer is $y = \mathrm{exec}(z, s)$. Methods differ in which of these pieces they model explicitly: an end-to-end network learns $p_\theta(y \mid I, q)$ directly; a module network builds a network from $z$; a neuro-symbolic system predicts $\hat z$ and $\hat s$ and runs a symbolic executor. Two properties are tested beyond accuracy:

• Compositional generalization: answering correctly when familiar attributes, objects or operations appear in combinations absent from training (CLEVR-CoGenT, below).
• Consistency: giving answers that agree across related questions, or across the images a sentence is paired with (GQA and NLVR2, below).

Input and Output

Benchmark	Input	Output
CLEVR, GQA	one image and a question	one answer from a fixed answer set
NLVR2	a pair of photographs and a sentence	true or false
SNLI-VE	an image (premise) and a sentence (hypothesis)	entailment, neutral or contradiction
VCR	an image, object detections and a question that refers to them	one of four answers, then one of four rationales
Winoground	two images and two captions with the same words in a different order	the correct image–caption pairing

Images range from rendered scenes of simple 3D shapes (CLEVR) to photographs (GQA, NLVR2, SNLI-VE, Winoground) and movie frames (VCR). Queries are generated from templates (CLEVR, GQA) or written by people (NLVR2, VCR, Winoground, CLEVR-Humans).

Variants

• Compositional question answering over synthetic scenes (CLEVR) and real images with scene graphs (GQA).
• Natural-language verification: deciding whether a sentence is true of an image or image pair (NLVR2), and visual entailment, a three-way version with a neutral class (SNLI-VE).
• Commonsense reasoning: inferring intentions, causes and what happens next, and justifying the answer (VCR).
• Compositional image–text matching: telling apart captions that differ only in word order or in a preposition (Winoground; What’sUp, by Kamath et al., 2023). It shares the scoring of image–text retrieval but uses minimal pairs.
• Compositional generalization tests: held-out attribute combinations (CLEVR-CoGenT), longer programs than in training, and human-written questions over synthetic images (CLEVR-Humans; Johnson et al., 2017, IEP).

Localizing the object a phrase refers to is visual grounding; several reasoning methods call a grounding model as one step. Reasoning over charts and document pages is part of document understanding.

Challenges

• Shortcuts. A learner can answer from the question alone or from correlations in the training data. CLEVR’s authors compare this to Clever Hans, the horse that appeared to do arithmetic by reading cues from its audience, and design the dataset so that such cues are scarce.
• Long reasoning chains. On CLEVR query questions, the error rate of every model they tested grew with the question’s effective size, the smallest program that gives the same answer on that scene, but not with its actual length: many long questions can be answered without resolving every reference (Johnson et al., 2017, CLEVR).
• Memory and comparison. All CLEVR baselines scored near 50% on attribute-comparison questions (“is the cube the same size as the sphere?”), although the best of them could query each attribute well; the authors read this as an inability to hold two objects in mind.
• Spatial semantics. Many CLEVR relational questions can be answered from absolute image position (“in the front half of the image” in place of “in front of the yellow metal sphere”): once those were excluded, no model beat the question-only LSTM on count questions or the question-type prior on existence questions, and the authors concluded that the models had not learned the semantics of spatial relations.
• Entangled attributes. Trained on one set of colour–shape pairings, CLEVR models learned strong biases about the colours of cubes and cylinders and answered colour questions about them much worse when the pairings changed (CLEVR-CoGenT).
• Perception as the bottleneck on real images. On Winoground’s photographs, small or out-of-focus objects and unusual images or captions account for part of the failures that look like failures of compositional reasoning (Diwan et al., 2022).

Approaches

End-to-end baselines and attention

CLEVR’s authors reproduced the VQA models of 2016: question-only baselines (the most frequent answer per question type; an LSTM), CNN features concatenated with a bag of words or an LSTM encoding, compact bilinear pooling (MCB) and stacked spatial attention (SA), all on frozen ImageNet ResNet-101 features. The best, CNN+LSTM+SA, reached 68.5% against 92.6% for people; the question-only LSTM reached 46.8% (Johnson et al., 2017, CLEVR, Figure 4). CNN+BoW was at chance on integer comparisons because averaging word vectors mixes the descriptions of the two sets being compared. Attention helped attribute queries, which need one object, but not attribute comparisons, and helped counting only a little (52.2% against 43.7% for CNN+LSTM).

Neural module networks

Neural module networks (NMN; Andreas et al., 2016) assemble a different network for each question from a parse: attention modules such as find[red] and transform[above] feed a measure or describe module that outputs the answer, and the modules are trained jointly from answers alone. The VQA article describes the modules and their VQA result. To test composition directly, Andreas et al. built SHAPES, 244 unique yes/no questions about arrangements of coloured shapes, each paired with 64 images. NMN reached 90.6% against 65.3% for a VIS+LSTM baseline and 63.0% for always giving the majority answer. Trained without any size-6 questions (networks of six modules), it scored 89.7% on them, against 85.2% when they were in training (Table 2, row “NMN (train size ≤ 5)”; the caption calls this variant “NMN (easy)”). The approach depended on an off-the-shelf parser to produce layouts, which Johnson et al. found failed on CLEVR’s complex questions (IEP).

Program generator and execution engine

Johnson et al.’s second 2017 paper, Inferring and Executing Programs for Visual Reasoning (IEP), learns both halves. A sequence-to-sequence LSTM, the program generator, maps the question to a program in prefix notation; the execution engine turns each function into a generic module (a residual block over $C \times H \times W$ feature maps; binary modules first concatenate their inputs) and assembles them into a network over ResNet-101 conv4 features. In the semi-supervised setting, training first fits the generator to a few ground-truth programs, then trains the engine on predicted programs, then fine-tunes both jointly with REINFORCE using answer correctness as reward. With all 700K programs it reached 96.9% on CLEVR test; with 18K it reached 95.4%, and with 9K 88.6% (Table 1). The authors note that CLEVR has about 450K unique programs, so 18K programs is at most 4% of them. On CLEVR-Humans, 17,817 training, 7,202 validation and 7,145 test questions written by crowd workers about CLEVR images, it scored 54.0% before and 66.6% after fine-tuning only the program generator (Table 3). The baseline numbers in IEP’s Table 1 differ from those in the CLEVR paper: CNN+LSTM+SA scores 69.8% and Q-type mode 42.1%, against 68.5% and 41.8% in the CLEVR paper’s Figure 4.

FiLM and MAC

FiLM (Perez et al., 2018) drops explicit programs. A GRU reads the question and predicts a per-channel scale and shift, $\gamma \odot F + \beta$, for each of four residual blocks of a CNN over the image, with coordinate maps concatenated to give positions; Multimodal Fusion explains the operator and its CLEVR result. Trained only on image–question–answer triples, it reached 97.7% on CLEVR with frozen ResNet-101 features and 97.6% from raw pixels (Table 1). The MAC network (Memory, Attention and Composition; Hudson & Manning, 2018), which Suhr et al. (2019) describe as a sequence of attention-based steps and GQA’s authors as a compositional attention model for CLEVR, was not read for this article; its results here are those listed in later papers: 98.9% on CLEVR as listed by Yi et al. (2018), and 54.06% on GQA test in GQA’s own table.

Neuro-symbolic execution

NS-VQA (Yi et al., 2018) separates perception, language and reasoning completely. A Mask R-CNN proposes object segments and predicts each one’s colour, material, size and shape, and a ResNet-34 regresses its 3D position, giving a structural scene representation, a table of objects and attributes. An attention-based sequence-to-sequence model translates the question into a program, and a deterministic executor written in Python runs the program on the table. The question parser is pretrained on a few ground-truth programs and fine-tuned with REINFORCE on answers. With 270 programs (three per CLEVR question family) and 4,000 images to train the scene parser, NS-VQA reported 99.8% on CLEVR; with 90 programs, 74.4% (Table 1).

Explicit structure does not by itself guarantee compositional generalization. On CLEVR-CoGenT, trained on condition A, NS-VQA scored 99.8% on A but 63.9% on B, lower than every other model in the authors’ table, because its attribute network had learned to classify shape partly from colour (Table 2a). A separate shape network that sees only a grey-scale image (NS-VQA+Gray) restored B to 98.4% without any data from B. The authors attribute the failure to perception and report that the question parser and executor did not overfit to either split.

Program-generating language models

Later systems replace the trained program generator with a large language model prompted with examples. VisProg (Gupta & Kembhavi, 2023) prompts GPT-3 with a few examples of instructions and hand-written programs; the generated program calls 20 modules, among them OWL-ViT for open-vocabulary localization, a ViLT VQA model, CLIP, face detection, image cropping by spatial preposition, counting, Python expressions and Stable Diffusion for editing. Nothing is trained. On a subset of GQA testdev (up to 20 questions per question type), VisProg scored 50.5% with majority voting over five prompts of 24 examples, against 47.8% for the ViLT VQA model it calls; on the NLVR2 public test set it reached 62.4% zero-shot, against 76.3% for ViLT fine-tuned on NLVR2 (Tables 1–2). Its visual rationales, the intermediate outputs of each step, showed that incorrect programs, the leading source of error on GQA, affected 16% of about 100 inspected samples. The paper received a CVPR 2023 Best Paper award.

ViperGPT (Surís et al., 2023) gives a code model (Codex) an API specification, function signatures and docstrings without implementations, and lets it write ordinary Python, with loops, conditionals and libraries, that calls perception models such as GLIP, X-VLM, MiDaS depth and BLIP-2. On GQA test-dev it reported 48.1% zero-shot against 44.7% for BLIP-2 (Table 2; the text refers to the table as Table 4); supervised models in the same table reach 55.8–72.1%. Both systems inherit the accuracy of their modules: in our reading, they move the bottleneck from reasoning to perception, as NS-VQA’s CoGenT result did.

Pretrained transformers and vision–language models

Vision–language pretraining made NLVR2 a standard fine-tuning benchmark, with architectures adapted to take two images (the BLIP article describes one). Vision–language models answer GQA-style questions in free text, but scores are often not zero-shot: LLaVA-1.5’s instruction mixture includes GQA training images, as the instruction tuning article notes. Diagnostic benchmarks keep finding gaps:

• Word order. On Winoground, every one of the 17 model configurations evaluated (including UNITER, VinVL, ViLT, LXMERT and CLIP ViT-B/32) scored below the 16.67% chance level on the group score, against 85.5% for crowd workers; the best image score was 17.75% (VinVL) against 88.5% (Thrush et al., 2022, Table 3).
• Spatial prepositions. On What’sUp, photographs of the same two objects placed on, under, in front of, behind, left of or right of each other, and on spatial subsets of COCO and GQA, all 18 models evaluated performed poorly. BLIP fine-tuned on VQAv2 averaged 56% over the three benchmarks, against estimated human accuracy of 97.3–100%; the authors found little reliable spatial supervision in LAION-2B, and up-weighting preposition-containing captions or fine-tuning on their data did not close the gap (Kamath et al., 2023).
• Broad VLM benchmarks. MMBench’s authors found all models weak at spatial relations between objects, as the VLM article reports.

Results on CLEVR

Model	Supervision	CLEVR test accuracy (%)	Source
Q-type mode	answers	41.8	Johnson et al., 2017 (CLEVR), Fig. 4
LSTM (question only)	answers	46.8	same
CNN+LSTM+SA	answers	68.5	same
IEP, 9K / 18K / 700K programs	programs + answers	88.6 / 95.4 / 96.9	Johnson et al., 2017 (IEP), Table 1
FiLM (raw pixels)	answers	97.6	Perez et al., 2018, Table 1
FiLM (ResNet-101 features)	answers	97.7	same
MAC (as listed by Yi et al.)	answers	98.9	Yi et al., 2018, Table 1
NS-VQA, 270 programs	programs + answers + scene annotations	99.8	same
Humans (5,500 test questions)	—	92.6	Johnson et al., 2017 (CLEVR)

Supervision differs between rows: IEP and NS-VQA use program annotations, and NS-VQA’s scene parser is trained on ground-truth object attributes and positions from 4,000 images. With several systems near or above human accuracy, the harder tests are CoGenT, CLEVR-Humans and real-image benchmarks.

Results on CLEVR-CoGenT

Model	Supervision on A	Trained on A, tested on A	Trained on A, tested on B	Source
CNN+LSTM+SA+MLP	answers	80.3	68.7	Johnson et al., 2017 (IEP), Fig. 5
IEP	18K programs + answers	96.6	73.7	same
FiLM	answers	98.3	75.6	Perez et al., 2018, Fig. 9
NS-VQA	500 programs + answers; scene parser on 4,000 annotated A images	99.8	63.9	Yi et al., 2018, Table 2a
NS-VQA+Gray (grey-scale shape network)	same	99.6	98.4	same

Accuracy in percent on CoGenT’s condition A and B evaluation sets as printed (FiLM’s paper names validation B), with no data from B. Yi et al.’s table lists FiLM at 78.8 on B, which is FiLM’s “0-shot” row, obtained by combining FiLM parameters of related questions; FiLM’s plain row is 75.6. With 3K images and 30K questions of fine-tuning on B, the baselines recovered on B but lost accuracy on A (IEP: 92.7 on B, 76.1 on A); FiLM’s authors call the same loss in their own model catastrophic forgetting.

Example: a learned model and a symbolic program on held-out combinations

The script builds a small CLEVR-like world: $32 \times 32$ images of three to five non-overlapping red, green, blue or yellow circles, squares and triangles in two sizes, with pixel noise. Questions come from three program families with eight possible answers: count (“how many red triangles are there”, answers 0–2 after balancing), more (“are there more blue circles than green squares”) and relate (“is there a red triangle left of the yellow circle”, asked only when the second object is unique). As in CLEVR, every answer is computed by executing the program on the ground-truth scene, and rejection sampling pushes answers toward balance within each family (counts of 2 stay rarer than 0 and 1), so the most frequent training answer per family scores about 50% (printed). Four colour–shape combinations (blue and yellow squares, red and green triangles) are held out, in one of two senses, since the sources leave open which one a test should use:

• “questions”: the held-out combinations appear in training images but are never mentioned in training questions;
• “CoGenT”: as in CLEVR-CoGenT, training images come from condition A (squares only red or green, triangles only blue or yellow, circles any colour) and held-out questions are asked about condition B images, where the palettes are swapped.

Two systems answer the questions:

• Learned: a FiLM-style network trained end to end on (image, question, answer) triples: a three-layer CNN to an $8 \times 8$ grid, a GRU over the question tokens that predicts $(\gamma, \beta)$ for two residual blocks with coordinate maps, and a max-pooled classifier over the eight answers; 10 epochs of Adam on 16,000 questions, seeds 0, 1 and 2.
• Symbolic: a de-renderer in the spirit of NS-VQA. Connected components of the foreground give the objects; a logistic regression on each component’s mean RGB predicts its colour, and another on a $12 \times 12$ crop predicts its shape; both are trained on object crops from the training images. The question’s program, given as ground truth here (the step that IEP and NS-VQA learn), is executed on the predicted scene. Following NS-VQA’s grey-scale shape network, the shape classifier sees either the RGB crop or its grey-scale mean, and the script prints both.

The script uses PyTorch 2.14.1, NumPy 2.5.4, SciPy 1.18.1 and scikit-learn 1.9.1, runs in about 4.5 minutes on one CPU thread, most of it training the learned model six times, and printed identical output on two runs. The plan sized it from a timed first epoch: the first planned version (48-pixel images, 24,000 questions, 15 epochs) would have taken about half an hour, so images, object sizes, model width, training set and epochs were reduced before the first run.

import sys
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from scipy import ndimage
from sklearn.linear_model import LogisticRegression

torch.set_num_threads(1)
S = 32
COLORS = {"red": (1, .2, .2), "green": (.2, 1, .2), "blue": (.2, .2, 1), "yellow": (1, 1, .2)}
CNAMES, SHAPES = list(COLORS), ["circle", "square", "triangle"]
RGB = np.array(list(COLORS.values()))
HELD = {("blue", "square"), ("yellow", "square"), ("red", "triangle"), ("green", "triangle")}
ALLOWED = {  # colours allowed per shape in each image condition (CoGenT-style)
    "all": {s: CNAMES for s in SHAPES},
    "A": {"circle": CNAMES, "square": ["red", "green"], "triangle": ["blue", "yellow"]},
    "B": {"circle": CNAMES, "square": ["blue", "yellow"], "triangle": ["red", "green"]},
}
RELS = {"left": lambda o, r: o["x"] < r["x"], "right": lambda o, r: o["x"] > r["x"],
        "above": lambda o, r: o["y"] < r["y"], "below": lambda o, r: o["y"] > r["y"]}
ANSWERS = [str(i) for i in range(6)] + ["yes", "no"]
VOCAB = ["<pad>", "how", "many", "are", "there", "more", "than", "is", "a", "the", "of"] \
    + CNAMES + SHAPES + list(RELS)
yy, xx = np.mgrid[0:S, 0:S] + 0.5


def make_scene(cond, rng):
    """3-5 non-overlapping objects; centres differ by >= 3 px in x and in y."""
    objs = []
    while len(objs) < rng.integers(3, 6):
        s = SHAPES[rng.integers(3)]
        c = ALLOWED[cond][s][rng.integers(len(ALLOWED[cond][s]))]
        r = (3, 5)[rng.integers(2)]
        x, y = rng.uniform(r + 1, S - r - 1, size=2)
        if all(np.hypot(x - o["x"], y - o["y"]) >= r + o["r"] + 2 and abs(x - o["x"]) >= 3
               and abs(y - o["y"]) >= 3 for o in objs):
            objs.append(dict(color=c, shape=s, r=r, x=x, y=y))
    return objs


def render(objs, rng):
    img = np.full((S, S, 3), 0.1)
    for o in objs:
        dx, dy, r = xx - o["x"], yy - o["y"], o["r"]
        m = {"circle": dx**2 + dy**2 < r**2,
             "square": (abs(dx) < 0.85 * r) & (abs(dy) < 0.85 * r),
             "triangle": (dy < 0.8 * r) & (dy > -r + 2 * abs(dx))}[o["shape"]]
        img[m] = RGB[CNAMES.index(o["color"])] * rng.uniform(0.8, 1.0)
    return np.clip(img + 0.05 * rng.standard_normal(img.shape), 0, 1)


# ---------- programs: each question is a small functional program ----------
def count(scene, c, s):
    return sum(o["color"] == c and o["shape"] == s for o in scene)


def execute(prog, scene):
    if prog[0] == "count":
        return str(min(count(scene, *prog[1]), 5))
    if prog[0] == "more":
        return "yes" if count(scene, *prog[1]) > count(scene, *prog[2]) else "no"
    _, (c1, s1), rel, (c2, s2) = prog            # "relate": is there a c1 s1 <rel> the c2 s2?
    ref = [o for o in scene if o["color"] == c2 and o["shape"] == s2]
    if len(ref) != 1:
        return None                               # ill-posed: referent not unique
    return "yes" if any(o is not ref[0] and o["color"] == c1 and o["shape"] == s1
                        and RELS[rel](o, ref[0]) for o in scene) else "no"


def tokens(prog):
    if prog[0] == "count":
        w = ["how", "many", *prog[1], "are", "there"]
    elif prog[0] == "more":
        w = ["are", "there", "more", *prog[1], "than", *prog[2]]
    else:
        w = ["is", "there", "a", *prog[1], prog[2], "of", "the", *prog[3]]
    ids = [VOCAB.index(t) for t in w]
    return [0] * (10 - len(ids)) + ids            # left-pad to 10 tokens


ALL_COMBOS = [(c, s) for c in CNAMES for s in SHAPES]


def sample_question(scene, mode, rng):
    """mode 'seen': no mentioned combo is held out; 'held': at least one is."""
    ok = lambda combos: (any(k in HELD for k in combos) if mode == "held"
                         else not any(k in HELD for k in combos))
    present = [(o["color"], o["shape"]) for o in scene]
    pick = lambda: (present[rng.integers(len(present))] if rng.random() < 0.75
                    else ALL_COMBOS[rng.integers(12)])
    fam = rng.integers(3)
    want = ("yes", "no")[rng.integers(2)] if fam else str(rng.integers(3))   # balanced targets
    for _ in range(200):
        if fam == 0:
            prog = ("count", pick())
        elif fam == 1:
            prog = ("more", pick(), pick())
            if prog[1] == prog[2]:
                continue
        else:
            prog = ("relate", pick(), list(RELS)[rng.integers(4)], pick())
        combos = [p for p in prog[1:] if isinstance(p, tuple)]
        a = execute(prog, scene)
        if a is None or not ok(combos):
            continue
        if a == want:
            return prog, a
    return None


def make_set(n_img, q_per_img, cond, mode, rng):
    scenes, imgs, qs, progs, ans, img_idx = [], [], [], [], [], []
    while len(scenes) < n_img:
        sc = make_scene(cond, rng)
        got = [sample_question(sc, mode, rng) for _ in range(q_per_img)]
        if any(g is None for g in got):
            continue
        for p, a in got:
            progs.append(p); qs.append(tokens(p)); ans.append(ANSWERS.index(a)); img_idx.append(len(scenes))
        scenes.append(sc); imgs.append(render(sc, rng))
    return dict(scenes=scenes, imgs=np.array(imgs, np.float32), q=np.array(qs), a=np.array(ans),
                prog=progs, idx=np.array(img_idx))


# ---------- learned model: CNN + GRU question encoder + FiLM ----------
class FiLMNet(nn.Module):
    def __init__(self, ch=16):
        super().__init__()
        self.stem = nn.Sequential(nn.Conv2d(3, ch, 3, 2, 1), nn.ReLU(), nn.Conv2d(ch, ch, 3, 2, 1), nn.ReLU(),
                                  nn.Conv2d(ch, ch, 3, 1, 1), nn.ReLU())        # 32 -> 8 x 8
        self.emb, self.gru = nn.Embedding(len(VOCAB), 32), nn.GRU(32, 64, batch_first=True)
        self.film = nn.Linear(64, 2 * 2 * ch)                                    # gamma, beta for 2 blocks
        self.c1 = nn.ModuleList([nn.Conv2d(ch + 2, ch, 1) for _ in range(2)])
        self.c2 = nn.ModuleList([nn.Conv2d(ch, ch, 3, 1, 1) for _ in range(2)])
        self.bn = nn.ModuleList([nn.BatchNorm2d(ch, affine=False) for _ in range(2)])
        self.head = nn.Sequential(nn.Conv2d(ch + 2, 64, 1), nn.ReLU())
        self.mlp = nn.Sequential(nn.Linear(64, 64), nn.ReLU(), nn.Linear(64, len(ANSWERS)))
        g = torch.linspace(-1, 1, 8)
        self.register_buffer("coords", torch.stack(torch.meshgrid(g, g, indexing="ij"))[None])

    def forward(self, img, q):
        h = self.stem(img)
        coords = self.coords.expand(len(h), -1, -1, -1)
        gb = self.film(self.gru(self.emb(q))[1][0]).view(len(h), 2, 2, -1)
        for k in range(2):                                    # FiLM-ed residual blocks
            x = F.relu(self.c1[k](torch.cat([h, coords], 1)))
            x = self.bn[k](self.c2[k](x))
            x = F.relu((1 + gb[:, k, 0, :, None, None]) * x + gb[:, k, 1, :, None, None])
            h = h + x
        return self.mlp(self.head(torch.cat([h, coords], 1)).amax((2, 3)))


def train_learned(tr, seed, epochs):
    torch.manual_seed(seed)
    net = FiLMNet()
    opt = torch.optim.Adam(net.parameters(), 1e-3)
    imgs = torch.tensor(tr["imgs"]).permute(0, 3, 1, 2)
    q, a, idx = torch.tensor(tr["q"]), torch.tensor(tr["a"]), torch.tensor(tr["idx"])
    g = torch.Generator().manual_seed(seed)
    for _ in range(epochs):
        net.train()
        for b in torch.randperm(len(a), generator=g).split(64):
            loss = F.cross_entropy(net(imgs[idx[b]], q[b]), a[b])
            opt.zero_grad(); loss.backward(); opt.step()
    return net


@torch.no_grad()
def acc_learned(net, ts):
    net.eval()
    imgs = torch.tensor(ts["imgs"]).permute(0, 3, 1, 2)
    pred = net(imgs[torch.tensor(ts["idx"])], torch.tensor(ts["q"])).argmax(1).numpy()
    return (pred == ts["a"]).mean()


# ---------- symbolic pipeline: perceive a scene, then execute the program ----------
class Region:
    def __init__(self, coords):
        self.coords, self.area = coords, len(coords)
        self.centroid = coords.mean(0)
        self.bbox = (*coords.min(0), *(coords.max(0) + 1))


def segment(img):
    """Connected components (8-connectivity) of the foreground; tiny ones dropped."""
    lab, n = ndimage.label((img > 0.3).any(2), structure=np.ones((3, 3)))
    regions = [Region(np.argwhere(lab == k)) for k in range(1, n + 1)]
    return [p for p in regions if p.area >= 8]


def features(img, p, grey):
    """Colour: mean RGB of the component. Shape: its square crop resampled to 12 x 12."""
    r0, c0, r1, c1 = p.bbox
    side = max(r1 - r0, c1 - c0)
    cy, cx = (r0 + r1 - 1) / 2, (c0 + c1 - 1) / 2
    g = np.linspace(-(side - 1) / 2, (side - 1) / 2, 12)
    rows, cols = np.meshgrid(cy + g, cx + g, indexing="ij")
    crop = np.stack([ndimage.map_coordinates(img[..., k], [rows, cols], order=1, cval=0.1)
                     for k in range(3)], -1)
    shape_f = crop.mean(2).ravel() if grey else crop.ravel()
    color_f = img[p.coords[:, 0], p.coords[:, 1]].mean(0)
    return color_f, shape_f


def fit_perception(tr, grey):
    Xc, Xs, yc, ys = [], [], [], []
    for img, sc in zip(tr["imgs"], tr["scenes"]):
        for p in segment(img):
            cy, cx = p.centroid
            d = [np.hypot(o["x"] - 0.5 - cx, o["y"] - 0.5 - cy) for o in sc]
            o = sc[int(np.argmin(d))]
            if min(d) > 3:
                continue
            cf, sf = features(img, p, grey)
            Xc.append(cf); Xs.append(sf); yc.append(o["color"]); ys.append(o["shape"])
    col = LogisticRegression(max_iter=2000).fit(Xc, yc)
    shp = LogisticRegression(max_iter=2000).fit(Xs, ys)
    return col, shp


def perceive(img, col, shp, grey):
    scene = []
    for p in segment(img):
        cf, sf = features(img, p, grey)
        cy, cx = p.centroid
        scene.append(dict(color=col.predict([cf])[0], shape=shp.predict([sf])[0], x=cx, y=cy))
    return scene


def acc_symbolic(ts, col, shp, grey):
    scenes = [perceive(img, col, shp, grey) for img in ts["imgs"]]
    pred = [execute(p, scenes[i]) for p, i in zip(ts["prog"], ts["idx"])]
    return np.mean([pa == ANSWERS[a] for pa, a in zip(pred, ts["a"])])


def shape_acc_on_held(ts, shp, grey):
    """Shape accuracy of the perception head on objects whose combo is held out."""
    hit = []
    for img, sc in zip(ts["imgs"], ts["scenes"]):
        for p in segment(img):
            cy, cx = p.centroid
            d = [np.hypot(o["x"] - 0.5 - cx, o["y"] - 0.5 - cy) for o in sc]
            o = sc[int(np.argmin(d))]
            if min(d) <= 3 and (o["color"], o["shape"]) in HELD:
                hit.append(shp.predict([features(img, p, grey)[1]])[0] == o["shape"])
    return np.mean(hit)


SPLITS = {  # name: (train condition, in-distribution test condition, held-out test condition)
    "questions": ("all", "all", "all"),
    "CoGenT": ("A", "A", "B"),
}

if __name__ == "__main__":
    epochs = int(sys.argv[1]) if len(sys.argv) > 1 else 10
    seeds = [0, 1, 2]
    for split, (c_tr, c_in, c_ho) in SPLITS.items():
        rng = np.random.default_rng(0)
        tr = make_set(4000, 4, c_tr, "seen", rng)
        t_in = make_set(1000, 2, c_in, "seen", rng)
        t_ho = make_set(1000, 2, c_ho, "held", rng)
        print(f"split '{split}': {len(tr['a'])} training questions; 2,000 test questions each")
        fam = lambda d: np.array([p[0] for p in d["prog"]])
        maj = {f: np.bincount(tr["a"][fam(tr) == f]).argmax() for f in ("count", "more", "relate")}
        base = [np.mean([maj[f] == y for f, y in zip(fam(d), d["a"])]) for d in (t_in, t_ho)]
        print(f"  majority answer per family:        in-dist {base[0]:.3f}         held-out {base[1]:.3f}")
        acc = np.array([[acc_learned(net, t_in), acc_learned(net, t_ho)]
                        for net in (train_learned(tr, s, epochs) for s in seeds)])
        print(f"  learned (FiLM), seeds {seeds}: in-dist {acc[:, 0].min():.3f}-{acc[:, 0].max():.3f}   "
              f"held-out {acc[:, 1].min():.3f}-{acc[:, 1].max():.3f}")
        for grey in (False, True):
            col, shp = fit_perception(tr, grey)
            name = "grey" if grey else "RGB"
            print(f"  symbolic, {name:4s} shape head:    in-dist {acc_symbolic(t_in, col, shp, grey):.3f}         "
                  f"held-out {acc_symbolic(t_ho, col, shp, grey):.3f}   "
                  f"(shape acc. on held-out-combo objects {shape_acc_on_held(t_ho, shp, grey):.3f})")

split 'questions': 16000 training questions; 2,000 test questions each
  majority answer per family:        in-dist 0.507         held-out 0.544
  learned (FiLM), seeds [0, 1, 2]: in-dist 0.624-0.633   held-out 0.568-0.594
  symbolic, RGB  shape head:    in-dist 0.997         held-out 1.000   (shape acc. on held-out-combo objects 1.000)
  symbolic, grey shape head:    in-dist 0.998         held-out 1.000   (shape acc. on held-out-combo objects 1.000)
split 'CoGenT': 16000 training questions; 2,000 test questions each
  majority answer per family:        in-dist 0.480         held-out 0.470
  learned (FiLM), seeds [0, 1, 2]: in-dist 0.651-0.679   held-out 0.518-0.563
  symbolic, RGB  shape head:    in-dist 0.998         held-out 0.798   (shape acc. on held-out-combo objects 0.748)
  symbolic, grey shape head:    in-dist 0.997         held-out 1.000   (shape acc. on held-out-combo objects 1.000)

The symbolic pipeline answers nearly every question in both splits, with one exception: under the CoGenT split its RGB shape classifier labels only 74.8% of the held-out-combination objects correctly, and accuracy on held-out questions falls from 99.8% to 79.8%. Trained on crops in which every triangle was blue or yellow, a classifier on raw colour pixels can use colour as evidence for shape; the grey-scale classifier cannot, and it recognizes every held-out object. This is NS-VQA’s CoGenT failure and fix in miniature. In the “questions” split the classifiers saw every combination during training, so nothing is held out from perception, and the program executor has no notion of which combinations were asked about: the held-out questions are new only to the language side, which here is given.

The learned model is the weaker system even in distribution: 62–68% against a family-majority baseline of 48–51%, so with 16,000 questions and 10 epochs it learned only part of the task; training accuracy is not printed, and in our reading the small training set is the likely cause (CLEVR’s models train on about 700,000 questions). Across the three seeds, its held-out range lies wholly below its in-distribution range in both splits, and under CoGenT its held-out accuracy is only 5–9 points above the 47.0% family-majority baseline. The example therefore shows the direction of the CLEVR-CoGenT results, but not their scale: a model that has not learned the task in distribution cannot show how well a well-trained network generalizes. What it does show is the property that separates the two systems. In our reading, a modular system generalizes where each module’s inputs stay inside what that module was trained on (colour and shape separately), and fails exactly where a module can exploit a correlation that the test breaks.

Datasets and Benchmarks

Dataset	Year	Images	Queries	Task and split notes
SHAPES	2016	synthetic shape arrangements	244 unique questions × 64 images (14,592 train, 1,024 test)	yes/no
CLEVR	2017	100,000 rendered (70K/15K/15K)	999,968 (853,554 unique), 90 question families	answer classification; programs and scene graphs provided
CLEVR-CoGenT	2017	rendered, conditions A and B	CLEVR-style	train on A, test on A and B
CLEVR-Humans	2017	CLEVR images	17,817 / 7,202 / 7,145 human-written	one question per image, CLEVR answers
GQA	2019	113K (Visual Genome)	22M generated; 1.7M balanced	70% train, 10% val, 10% test, 10% challenge, split by image
NLVR2	2019	127,502 web photographs	107,292 examples, 29,680 unique sentences	true/false on image pairs; dev, Test-P (public), Test-U (unreleased)
SNLI-VE	2019	31,783 Flickr30K images, disjoint across splits	529,527 / 17,858 / 17,901 pairs	entailment, neutral, contradiction
VCR	2019	110K movie scenes	290K multiple-choice problems	Q→A, QA→R, Q→AR
Winoground	2022	800	800 captions in 400 examples	text, image and group scores; evaluation only

• CLEVR (Johnson et al., 2017) renders three to ten objects with three shapes, two sizes, two materials and eight colours, related by left, right, behind and in front (defined by the camera’s viewing direction projected onto the ground plane) and by same-attribute relations. Questions are generated from templates of functional programs by a depth-first search that prunes ill-posed questions (a reference with no unique object) and degenerate ones (a reference that is unnecessary), and rejection sampling makes answers roughly uniform within each question family. On attribute queries the question-only baselines scored near chance (about 50%, 12.5%, 50% and 33.3% for size, colour, material and shape), showing little question-conditional bias; counts, ranging from 0 to 10, could only be pushed toward uniform. The authors warn that a hand-crafted system with knowledge of the CLEVR world could do well without generalizing, so the dataset is meant to be used together with real-image benchmarks. CoGenT keeps spheres in all colours but gives cubes and cylinders disjoint four-colour palettes in conditions A and B.
• GQA (Hudson & Manning, 2019) normalizes the Visual Genome scene graphs of 113K images (extended with 5K hidden scene graphs for testing) and generates questions from 524 structural patterns in 117 question groups, each group with a functional program. Balancing smooths the answer distribution within groups of questions at two granularities, while keeping real-world tendencies “up to a tunable degree”, and reduces 22M questions to 1.7M. Each question has a structural type (verify, query, choose, logical, compare) and a semantic type (object, attribute, category, relation, global).
• NLVR2 (Suhr et al., 2019) asked crowd workers to write one sentence that is true for two of four pairs of similar photographs and false for the other two, so each sentence appears with both labels and cannot be judged from the text alone. A text-only model scored exactly the majority rate (51.1% on Test-P), as the balancing guarantees. The best baseline, a maximum-entropy classifier over detected object counts, reached 54.8% on Test-P; the visual reasoning methods FiLM, N2NMN and MAC stayed near chance (52.1%, 51.1% and 51.4% on Test-P), most of them after fitting the training data far better, against human accuracy of 96.1–96.3% (Table 6).
• SNLI-VE is built from the SNLI sentence-pair corpus and Flickr30K: the premise is an image and the hypothesis a sentence, labelled entailment if the image gives enough evidence that the sentence is true, contradiction if it gives enough evidence that it is false, and neutral otherwise; the three labels are about evenly distributed in every split. The official repository reports about 9.2% incorrect labels in a sample of 217 pairs, mostly in the neutral class, and suggests dropping that class when label quality matters; a later relabelling (e-SNLI-VE-2.0) corrected neutral labels in dev and test. The task paper (Xie et al., 2019) is on arXiv only and was not read; the description here follows the repository.
• VCR (Zellers et al., 2019) collected questions, answers and rationales about movie frames, written by crowd workers who refer to detected people and objects by tags such as [person1]. Its wrong choices come from Adversarial Matching: correct responses written for other questions are assigned as wrong choices by maximum-weight bipartite matching, so that each is relevant to its new question (scored by BERT) but not too similar to that question’s correct response (scored by ESIM+ELMo). The data are split by movie into 11 folds, matched within each fold, with two folds held out for validation and testing.
• Winoground (Thrush et al., 2022) has 400 hand-curated examples, each two images and two captions containing the same words in a different order, such as “some plants surrounding a lightbulb” and “a lightbulb surrounding some plants”. Every example carries a linguistic tag for what is swapped (objects, relations, or both) and some carry visual tags (symbolic, series, pragmatics).

Evaluation Metrics

Plain accuracy on exact answers is the main metric for CLEVR, GQA, NLVR2 and SNLI-VE; unlike VQA accuracy, CLEVR and GQA have one correct answer per question. Diagnostic benchmarks report it per question type or per program property.

GQA’s additional metrics (Hudson & Manning, 2019):

• Consistency: for each question a model answers correctly, its accuracy on the questions entailed by it (derived from the programs), averaged over those questions.
• Validity and plausibility: whether the answer lies in the question’s scope (a colour for a colour question) and whether it occurs with the question’s subject anywhere in the dataset.
• Distribution: a chi-square distance between the true and predicted answer distributions (lower is better).
• Grounding: for attention models, the attention mass on the region the question or answer refers to.

In GQA’s test table, MAC scored 54.06% accuracy and 81.59% consistency, against 89.3% and 98.4% for people; a blind LSTM scored 41.07%. The abstract gives 42.1% for the blind LSTM, which differs from the table, and 54.1% for strong VQA models.

NLVR2 consistency is the share of unique sentences for which a model is correct on every image pair the sentence comes with (Suhr et al., 2019). Because each sentence appears with both labels, a model that judges from the sentence alone scores near zero: the majority baseline has 4.2% consistency on Test-P.

VCR scores Q→A (choose the answer among four), QA→R (given the correct answer, choose the rationale among four) and Q→AR, which counts an example correct only if both choices are right; chance is 25%, 25% and 6.25%. On the test set, the authors’ R2C model reached 65.1%, 67.3% and 44.0%, and a text-only BERT 53.9%, 64.5% and 35.0%; the table gives people 91.0%, 93.0% and 85.0%, without a split (Zellers et al., 2019, Table 1). In our reading, the strong text-only result shows that the choices still carry linguistic cues.

Winoground’s scores compare a model’s image–caption scores $s(C, I)$ within each example $(C_0, I_0, C_1, I_1)$:

$$\text{text} = [\,s(C_0, I_0) > s(C_1, I_0)\,] \wedge [\,s(C_1, I_1) > s(C_0, I_1)\,],$$ $$\text{image} = [\,s(C_0, I_0) > s(C_0, I_1)\,] \wedge [\,s(C_1, I_1) > s(C_1, I_0)\,],$$

and the group score requires both (Thrush et al., 2022). The paper gives random chance as 25%, 25% and 16.67%; by our count, two independent comparisons both hold with probability 1/4, and all four hold when $s(C_0, I_0)$ and $s(C_1, I_1)$ are the two largest scores, 4 of the 24 equally likely orderings. Diwan et al. (2022) re-tagged the examples and found that many test more than compositional language: 38 of 400 need a subtle visual detail (CLIP’s image and group scores were 0 on them), 56 images and 50 captions are unusual, and 171 examples remain as a cleaner test of compositionality, on which models were still poor.

Compositional generalization is reported as the drop from the training distribution to the shifted one, such as CoGenT’s A→A against A→B, and for module and program methods also as accuracy against the number of program annotations used.

Applications

• Evaluating vision–language models. The benchmarks above, especially GQA, NLVR2 and Winoground, are routine parts of VLM and pretraining evaluations, and probe abilities that broad benchmarks average away.
• Interpretable question answering. Program-based systems expose their intermediate steps: VisProg’s visual rationales were used to locate failing modules and correct instructions (Gupta & Kembhavi, 2023), and ViperGPT measured how much accuracy drops when a module is replaced by an uninformative default (Surís et al., 2023).
• Robotics and assistance are named as motivations: IEP cites robotic navigation, autonomous driving and surveillance, and NLVR2 robotic applications, as settings that need compositional queries. The embodied AI article covers how language-conditioned agents are evaluated.
• Image editing and tagging by instruction: VisProg composes the same modules for natural-language image editing and knowledge-based object tagging.

Open Problems

• Compositional generalization at scale. Every model family in the CoGenT table lost accuracy on condition B; in our reading, NS-VQA’s recovery came from a perception change chosen with knowledge of the shift.
• Perception versus reasoning. Near-perfect symbolic reasoning on CLEVR rests on near-perfect de-rendering of a simple world; on photographs, program-generating systems are bounded by their detectors and VQA modules, as VisProg’s and ViperGPT’s GQA numbers show.
• Word order and spatial relations in pretrained models. Winoground and What’sUp find chance-level behaviour on minimal pairs, and Kamath et al. trace part of it to scarce spatial supervision in web captions.
• Benchmarks that stay hard. CLEVR’s test set saturated; VCR’s choices are partly solvable from the text alone; Diwan et al. show that a reasoning benchmark can mix in perception and annotation difficulties. In our view, each benchmark answers a narrower question than its name suggests.
• Contamination. VLM training mixtures include GQA and other benchmarks’ training splits, so zero-shot claims need checking against the mixture.