Image Captioning

Image captioning is the task of generating a natural-language description of an image: given a photograph of a dog catching a frisbee, produce a sentence such as “a dog jumps to catch a frisbee in a park”. In the taxonomy of multimodal learning it is translation from images to text, and most captioners are encoder–decoders: an image encoder followed by a language decoder.

Problem Definition

Given an image $I$, produce a word sequence $S = (w_1, \dots, w_T)$ that a person would accept as a correct and relevant description. Neural captioners model the conditional distribution word by word,

$$p_\theta(S \mid I) = \prod_{t=1}^{T} p_\theta(w_t \mid w_{<t}, I),$$

and are trained by minimizing the cross-entropy of each reference word given the reference prefix, a scheme called teacher forcing:

$$\mathcal{L}_{\text{XE}}(\theta) = -\sum_{t=1}^{T} \log p_\theta(w^*_t \mid w^*_{<t}, I).$$

At test time the model searches for a high-probability sentence, $\hat{S} \approx \arg\max_S p_\theta(S \mid I)$.

Input and Output

The input is an RGB image; the output is a sentence over a fixed vocabulary (10,000 words in Show, Attend and Tell) ending in an end-of-sentence token. The image becomes a global vector, a grid of convolutional features, a set of detected-region features or Vision Transformer patch tokens.

Variants

• Dense captioning gives a box and a phrase for each of many regions; novel object captioning names objects absent from the caption training data (nocaps, below).
• Text-aware captioning reads text in the image: TextCaps has about 145,000 captions for 28,000 images (Sidorov et al., 2020).
• Paragraph, stylized, controllable and video captioning change the length, style, focus or input.

Challenges

• Many valid outputs: a correct caption can share few words with the references.
• Grounding: language priors push decoders toward plausible but absent content.
• Composition: “a red circle left of a blue square” and “a blue circle left of a red square” use the same words.
• Salience: people mention what matters, and choosing that is part of the task.

Approaches

Retrieval and templates

Farhadi et al. (2010) mapped images and sentences into a space of $\langle$object, action, scene$\rangle$ triplets and attached to an image the best-scoring sentence: captioning as retrieval, a framing that survives in image–text retrieval. Baby Talk (Kulkarni et al., 2011) combined object, attribute and spatial-relation detectors with text statistics in a conditional random field and filled templates, or an n-gram language model, with the predicted labels.

CNN encoder, recurrent decoder

Show and Tell (Vinyals et al., 2015) applied the sequence-to-sequence recipe of machine translation: an ImageNet-pretrained CNN encodes the image into a vector, fed once, as the first input, to an LSTM language model trained with $\mathcal{L}_{\text{XE}}$. Its ensemble reached BLEU-4 27.7 on 4,000 held-out COCO validation images. Concurrently, Karpathy and Fei-Fei (2015) aligned sentence fragments with image regions and generated descriptions of images and regions with a multimodal recurrent network; their COCO split, with 5,000 images each for validation and test, became the standard Karpathy split.

Attention over the image

A single vector must summarize everything a caption might mention. Show, Attend and Tell (Xu et al., 2015) kept the $14 \times 14 \times 512$ feature map of a VGG network as $L = 196$ annotation vectors $a_i$ and let the decoder recompute weights over them at every word:

$$e_{ti} = f_{\text{att}}(a_i, h_{t-1}), \qquad \alpha_{ti} = \frac{\exp(e_{ti})}{\sum_{k=1}^{L} \exp(e_{tk})}, \qquad \hat{z}_t = \sum_{i=1}^{L} \alpha_{ti} a_i,$$

with $f_{\text{att}}$ a small MLP of the region feature and the previous decoder state $h_{t-1}$. This soft attention is differentiable; hard attention samples a single location from $\alpha_t$ and is trained with a variational bound equivalent to REINFORCE. A “doubly stochastic” penalty $\lambda \sum_i (1 - \sum_t \alpha_{ti})^2$ pushes each location’s total attention over the caption toward 1, and the maps show where the model looked for each word.

Bottom-up and top-down attention (Anderson et al., 2018) changed what the decoder attends to: a Faster R-CNN with a ResNet-101 backbone, trained on Visual Genome object and attribute labels, proposes regions above a confidence threshold, up to 100 (a fixed 36 worked almost as well; both from the arXiv version’s supplement), each a mean-pooled 2,048-dimensional feature, and a top-down LSTM attends over these objects instead of a uniform grid. A single Up-Down model reached Karpathy-test CIDEr 113.5 with cross-entropy training and 120.1 after CIDEr optimization.

Optimizing the metric

Under teacher forcing the model never conditions on its own outputs, as it must at test time; Rennie et al. (2017), following Ranzato et al., call this mismatch exposure bias. The training loss is also not the evaluation metric. Self-critical sequence training (SCST; Rennie et al., 2017) treats the decoder as a policy and minimizes the negative expected reward $-\mathbb{E}_{w^s \sim p_\theta}[r(w^s)]$, with $r$ the CIDEr-D score of a sampled caption. It uses REINFORCE with the reward of the model’s own greedy caption $\hat{w}$ as the baseline:

$$\nabla_\theta \mathcal{L}(\theta) \approx -\big(r(w^s) - r(\hat{w})\big)\, \nabla_\theta \log p_\theta(w^s \mid I).$$

Samples that beat the greedy caption become more probable, with no learned critic. Starting from a cross-entropy model, SCST raised a single attention model’s Karpathy-test CIDEr from 99.4 to 114.0 (best of four seeds, beam search), and their ensemble scored 114.7 on the COCO server, against a previous best of 104.9. Optimizing CIDEr also lifted the other metrics; Up-Down and the Meshed-Memory Transformer later used the same two stages (the latter with an SCST variant).

Transformers and vision–language pretraining

Transformer captioners replaced the LSTM with attention layers. The Meshed-Memory Transformer (Cornia et al., 2020) added learned memory slots to self-attention over regions and connected every decoder layer to every encoder layer, reaching Karpathy-test CIDEr 131.2 with SCST. Vision–language pretraining then made captioning a fine-tuning task: Oscar added detected object tags as anchor words (Li et al., 2020), and BLIP trained an image-grounded decoder beside retrieval objectives (Li et al., 2022).

Vision–language models

Current vision–language models connect a pretrained image encoder to a large language model and treat captioning as one instruction among many; Vision–Language Model covers the architecture.

Results on COCO

Model	Year	Image input	Training	BLEU-4	METEOR	CIDEr	SPICE
Show and Tell (ensemble, 4K held-out COCO images)	2015	global CNN vector	XE	27.7	23.7	85.5	—
Show, Attend and Tell (hard attention)	2015	$14 \times 14$ VGG grid	XE	25.0	23.04	—	—
SCST, Att2all	2017	ResNet-101 grid	SCST	34.2	26.7	114.0	—
Up-Down	2018	Faster R-CNN regions	XE / SCST	36.2 / 36.3	27.0 / 27.7	113.5 / 120.1	20.3 / 21.4
M² Transformer	2020	regions	SCST	39.1	29.2	131.2	22.6
Oscar-Large	2020	regions + tags, pretrained	XE / SCST	37.4 / 41.7	30.7 / 30.6	127.8 / 140.0	23.5 / 24.5
BLIP, ViT-L, 129M images	2022	ViT patches, pretrained	XE	40.4	—	136.7	—

Results are single models on the COCO Karpathy test split as printed, unless the row says otherwise (XE: cross-entropy). Show, Attend and Tell computed BLEU without a brevity penalty; SCST’s figures are the best of four seeds. For scale, one human caption scored against an image’s other COCO references receives BLEU-4 21.7, METEOR 25.2 and CIDEr-D 85.4 (Chen et al., 2015). By our interpretation, models exceed these numbers by reproducing consensus phrasing, not by out-writing people.

Training and Decoding

Greedy decoding takes the most probable word at each step. Beam search keeps the $k$ most probable prefixes at every step, approximating $\arg\max_S p_\theta(S \mid I)$ more closely. Sampling draws each word from $p_\theta$, or from its most probable part (nucleus sampling), for diverse captions.

The script below (PyTorch 2.14.1, CPU, about 10 s) trains a Show-and-Tell-style captioner from scratch on $32 \times 32$ images of one or two colored shapes, with invented captions over a 12-token vocabulary that read left to right (“a red circle left of a blue square”). A small CNN’s $4 \times 4$ feature grid is flattened, since pooling would discard which shape is on the left, and fed once to a GRU decoder trained with teacher forcing for 300 steps. The 300 test images are the same for every seed, each with one reference.

import sys, math
from collections import Counter
import torch
from torch import nn
import torch.nn.functional as F

torch.set_num_threads(2)
SEED = int(sys.argv[1]) if len(sys.argv) > 1 else 4
STEPS = int(sys.argv[2]) if len(sys.argv) > 2 else 300
torch.manual_seed(SEED)

# Invented 12-token vocabulary: captions read left to right across the image.
VOCAB = ["<pad>", "<bos>", "<eos>", "a", "red", "green", "blue",
         "circle", "square", "triangle", "left", "of"]
IDX = {w: i for i, w in enumerate(VOCAB)}
COLORS = {"red": (1.0, 0.2, 0.2), "green": (0.2, 1.0, 0.2), "blue": (0.2, 0.2, 1.0)}
SHAPES = ["circle", "square", "triangle"]
yy, xx = torch.meshgrid(torch.arange(32.) + 0.5, torch.arange(32.) + 0.5, indexing="ij")

def draw(img, color, shape, cy, cx, r):
    y, x = yy - cy, xx - cx
    if shape == "circle":
        m = x**2 + y**2 < r**2
    elif shape == "square":
        m = (x.abs() < 0.85 * r) & (y.abs() < 0.85 * r)
    else:
        m = (y < 0.8 * r) & (y > -r + 2 * x.abs())
    img[:, m] = torch.tensor(COLORS[color])[:, None]

def make_example(g):
    """A noisy 32x32 RGB image with one or two shapes and its (invented) caption."""
    img = torch.full((3, 32, 32), 0.05)
    k = 1 + int(torch.rand(1, generator=g) < 0.7)              # 70% of images have two shapes
    cx = [8.0, 24.0] if k == 2 else [16.0]
    words = []
    for j in range(k):
        c = list(COLORS)[torch.randint(3, (1,), generator=g)]
        s = SHAPES[torch.randint(3, (1,), generator=g)]
        cy = 16 + 6 * (torch.rand(1, generator=g).item() - 0.5) * 2
        x0 = cx[j] + 2 * (torch.rand(1, generator=g).item() - 0.5) * 2
        draw(img, c, s, cy, x0, 4.5 + torch.rand(1, generator=g).item())
        words += (["left", "of"] if j == 1 else []) + ["a", c, s]
    img += 0.05 * torch.randn(img.shape, generator=g)
    return img, words

def batch(n, g):
    imgs, caps = zip(*[make_example(g) for _ in range(n)])
    T = max(len(c) for c in caps) + 2
    tok = torch.zeros(n, T, dtype=torch.long)                  # <pad> = 0
    for i, c in enumerate(caps):
        ids = [IDX["<bos>"]] + [IDX[w] for w in c] + [IDX["<eos>"]]
        tok[i, :len(ids)] = torch.tensor(ids)
    return torch.stack(imgs), tok, list(caps)

class Captioner(nn.Module):
    def __init__(self, d=128):
        super().__init__()
        self.cnn = nn.Sequential(                               # 32x32 -> 4x4 feature grid
            nn.Conv2d(3, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
            nn.Conv2d(32, 64, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
            nn.Conv2d(64, 64, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2))
        self.img2x = nn.Sequential(nn.Flatten(), nn.Linear(64 * 4 * 4, d), nn.LayerNorm(d))
        self.emb = nn.Embedding(len(VOCAB), d)
        self.gru = nn.GRU(d, d, batch_first=True)
        self.out = nn.Linear(d, len(VOCAB))

    def encode(self, x):                     # image fed once, as the first input (Show and Tell)
        v = self.img2x(self.cnn(x - 0.2))       # flatten, not pool: keeps left/right layout
        return self.gru(v[:, None], None)[1]                    # hidden state after the image

    def step(self, tokens, h):                                  # next-word logits
        o, h = self.gru(self.emb(tokens), h)
        return self.out(o), h

def train(model, steps=STEPS, bs=64):
    g = torch.Generator().manual_seed(1000 + SEED)
    opt = torch.optim.Adam(model.parameters(), lr=1e-3)
    for step in range(steps):
        x, tok, _ = batch(bs, g)
        logits, _ = model.step(tok[:, :-1], model.encode(x))   # teacher forcing: true prefix in
        loss = F.cross_entropy(logits.reshape(-1, len(VOCAB)), tok[:, 1:].reshape(-1),
                               ignore_index=0)
        opt.zero_grad(); loss.backward(); opt.step()
    return loss.item()

@torch.no_grad()
def greedy(model, x, max_len=10):
    h, tok, out = model.encode(x[None]), torch.tensor([[IDX["<bos>"]]]), []
    for _ in range(max_len):
        logits, h = model.step(tok, h)
        tok = logits[:, -1].argmax(-1, keepdim=True)
        if tok.item() == IDX["<eos>"]:
            break
        out.append(VOCAB[tok.item()])
    return out

@torch.no_grad()
def beam(model, x, k=3, max_len=10):
    beams = [(0.0, [IDX["<bos>"]], model.encode(x[None]))]     # (log prob, tokens, state)
    done = []
    for _ in range(max_len):
        cand = []
        for lp, seq, h in beams:
            logits, h2 = model.step(torch.tensor([[seq[-1]]]), h)
            logp = F.log_softmax(logits[0, -1], -1)
            top = logp.topk(k)
            for v, i in zip(top.values.tolist(), top.indices.tolist()):
                cand.append((lp + v, seq + [i], h2))
        cand.sort(key=lambda c: -c[0])
        beams = []
        for c in cand[:k]:                                      # finished beams leave the search
            (done if c[1][-1] == IDX["<eos>"] else beams).append(c)
        if not beams:
            break
    best = max(done or beams, key=lambda c: c[0])               # highest total log probability
    return [VOCAB[i] for i in best[1][1:] if i != IDX["<eos>"]]

def corpus_bleu4(cands, refs):
    """Corpus BLEU-4 with one reference per image (clipped counts, brevity penalty)."""
    match, total = [0] * 4, [0] * 4
    c_len = r_len = 0
    for c, r in zip(cands, refs):
        c_len, r_len = c_len + len(c), r_len + len(r)
        for n in range(1, 5):
            cn = Counter(tuple(c[i:i + n]) for i in range(len(c) - n + 1))
            rn = Counter(tuple(r[i:i + n]) for i in range(len(r) - n + 1))
            match[n - 1] += sum(min(k, rn[g]) for g, k in cn.items())
            total[n - 1] += max(len(c) - n + 1, 0)
    if min(match) == 0:
        return 0.0
    bp = 1.0 if c_len > r_len else math.exp(1 - r_len / c_len)
    return bp * math.exp(sum(math.log(m / t) for m, t in zip(match, total)) / 4)

model = Captioner()
final_loss = train(model)
x_test, _, refs = batch(300, torch.Generator().manual_seed(999))   # same test set for every seed
model.eval()
res = {}
for name, fn in [("greedy", greedy), ("beam-3", beam)]:
    hyps = [fn(model, x) for x in x_test]
    res[name] = (sum(h == r for h, r in zip(hyps, refs)) / len(refs), corpus_bleu4(hyps, refs), hyps)
print(f"seed {SEED}: final training loss {final_loss:.3f}")
for name, (em, b4, hyps) in res.items():
    wrong = Counter()                                           # wrong words, by kind
    for h, r in zip(hyps, refs):
        if len(h) != len(r):
            wrong["number of shapes"] += 1
        else:
            for a, b in zip(h, r):
                if a != b:
                    wrong["color" if b in COLORS else "shape" if b in SHAPES else "other"] += 1
    print(f"  {name}: exact match {em:.3f}, BLEU-4 {b4:.3f}, wrong words {dict(sorted(wrong.items()))}")
same = sum(a == b for a, b in zip(res["greedy"][2], res["beam-3"][2]))
print(f"  greedy and beam-3 captions identical on {same}/{len(refs)} test images")

seed 4: final training loss 0.162
  greedy: exact match 0.460, BLEU-4 0.815, wrong words {'color': 1, 'shape': 191}
  beam-3: exact match 0.463, BLEU-4 0.815, wrong words {'color': 1, 'shape': 191}
  greedy and beam-3 captions identical on 296/300 test images

Two runs with the same seed printed identical output. Over seeds 0–4, greedy exact match ranged from 41.3% to 56.3% and BLEU-4 from 0.771 to 0.849. Beam search changed at most 12 of the 300 captions and exact match by at most 0.7 points; by our reading, a confident decoder’s greedy choices already give nearly the most probable sentence. Almost every error was a shape, never the number of shapes. BLEU-4 stays near 0.8 while half the captions are wrong, since one wrong word leaves most n-grams intact.

Datasets and Benchmarks

• COCO Captions, with at least five captions per image, is the main benchmark; most papers report the Karpathy test split.
• Flickr8k and Flickr30k have about 8,000 and 31,000 images with five captions each (Karpathy and Fei-Fei, 2015).
• nocaps (Agrawal et al., 2019) has 166,100 captions for 15,100 Open Images validation and test images. Training uses COCO captions plus Open Images labels and boxes, so nearly 400 test classes have few or no training captions; results are split into in-domain, near-domain and out-of-domain.

Evaluation Metrics

Most automatic metrics compare a candidate caption $c$ with the references $S = \{s_1, \dots, s_m\}$ of its image; papers report scores multiplied by 100.

• BLEU-$N$ (Papineni et al., 2002), from machine translation, is $\text{BP} \cdot \exp\big(\sum_{n=1}^{N} \tfrac{1}{N} \log p_n\big)$: a geometric mean of n-gram precisions $p_n$, each candidate n-gram counting at most as often as in a single reference, times a brevity penalty $\text{BP} = \min(1, e^{1 - r/c})$ for candidate length $c$ and closest-reference length $r$, both summed over the corpus.
• METEOR aligns candidate and reference words by exact, stem, synonym and paraphrase matches and computes a weighted harmonic mean of precision and recall, reduced by a penalty for fragmented alignments (Chen et al., 2015).
• ROUGE-L is an F-measure on the longest common subsequence of candidate and reference, with $\beta = 1.2$ weighting recall (Chen et al., 2015).
• CIDEr (Vedantam et al., 2015), designed for captions, measures consensus. N-grams ($n = 1, \dots, 4$) are weighted by TF-IDF, with document frequency counted over the reference sets of all images, so words used for every image count little; CIDEr$_n$ averages the cosine similarity of the candidate’s TF-IDF vector $g^n(c)$ with each reference’s, and CIDEr averages over $n$. CIDEr-D, used by the COCO server, drops stemming, clips counts and adds a Gaussian length penalty ($\sigma = 6$) and a factor of 10:

$$\text{CIDEr-D}_n(c, S) = \frac{10}{m} \sum_{j=1}^{m} e^{-\frac{(l(c) - l(s_j))^2}{2\sigma^2}} \, \frac{\min\big(g^n(c), g^n(s_j)\big) \cdot g^n(s_j)}{\lVert g^n(c) \rVert \, \lVert g^n(s_j) \rVert}.$$

• SPICE (Anderson et al., 2016) parses candidate and references into scene graphs and computes an F-score over object, attribute and relation tuples such as (girl), (girl, young) and (girl, on-top-of, court), matching words through WordNet synonyms. It can be broken down by color, count or size.
• CLIPScore (Hessel et al., 2021) needs no references: it is $w \cdot \max(\cos(\mathbf{c}, \mathbf{v}), 0)$ with $w = 2.5$, where $\mathbf{c}$ and $\mathbf{v}$ are the CLIP embeddings of the caption (prefixed with “A photo depicts”) and of the image. RefCLIPScore takes the harmonic mean of this score and the caption’s best cosine similarity to a reference.
• CHAIR (Rohrbach et al., 2018) measures object hallucination over the 80 COCO object classes: CHAIR$_i$ is the fraction of mentioned objects that are not in the image according to its segmentations and reference captions, and CHAIR$_s$ the fraction of captions with at least one such object.

Human judgment and metric agreement. Show and Tell’s captions outscored human captions on BLEU-4 (27.7 against 21.7), yet raters judged them clearly worse (Vinyals et al., 2015). For the 15 entries of the 2015 COCO Captioning Challenge plus human captions, the system-level Pearson correlation with the share of captions judged as good as or better than a human one was 0.05 for BLEU-4, 0.15 for ROUGE-L, 0.43 for CIDEr, 0.53 for METEOR and 0.88 for SPICE (Anderson et al., 2016, arXiv version). CLIPScore correlated with human judgments better than reference-based metrics on several corpora (Hessel et al., 2021); on FOIL captions, in which one noun phrase of a COCO caption has been swapped for a wrong one, CLIPScore picked the true caption 87.2% of the time against 66.5% for BLEU-4 when each metric had one reference.

The script implements sentence-level BLEU and a simplified CIDEr-D in NumPy (2.3.5) and scores six candidates for a red circle left of a blue square, against invented references for six synthetic shape scenes. Like the COCO evaluation code, it weights raw n-gram counts by IDF; scores are on the raw scale, not multiplied by 100.

import numpy as np
from collections import Counter

# Invented reference captions for six synthetic shape scenes (four per image).
# Scene 0 is the test image: a red circle to the left of a blue square.
REFS = [
    ["a red circle to the left of a blue square", "a red circle and a blue square",
     "a small red circle left of a blue square", "a blue square to the right of a red circle"],
    ["a green triangle above a red square", "a red square below a green triangle",
     "a green triangle and a red square", "a triangle above a square"],
    ["a blue circle on a black background", "a single blue circle",
     "one blue circle in the middle", "a blue circle"],
    ["two green circles side by side", "a green circle next to a green circle",
     "two green circles on a black background", "a pair of green circles"],
    ["a yellow square to the right of a green triangle", "a green triangle and a yellow square",
     "a yellow square right of a green triangle", "a triangle to the left of a square"],
    ["a red triangle below a blue circle", "a blue circle above a red triangle",
     "a red triangle and a blue circle", "a circle above a triangle"],
]

def ngrams(words, n):
    return Counter(tuple(words[i:i + n]) for i in range(len(words) - n + 1))

def bleu(cand, refs, N=4):
    """Sentence-level BLEU-N: clipped n-gram precisions, geometric mean, brevity penalty."""
    c, rs = cand.split(), [r.split() for r in refs]
    logp = 0.0
    for n in range(1, N + 1):
        cnt = ngrams(c, n)
        max_ref = Counter()
        for r in rs:
            for g, k in ngrams(r, n).items():
                max_ref[g] = max(max_ref[g], k)
        clipped = sum(min(k, max_ref[g]) for g, k in cnt.items())
        if clipped == 0:
            return 0.0                                   # log 0: no smoothing
        logp += np.log(clipped / sum(cnt.values())) / N
    r = min((abs(len(x) - len(c)), len(x)) for x in rs)[1]   # closest reference length
    bp = 1.0 if len(c) > r else np.exp(1 - r / len(c))
    return bp * np.exp(logp)

def cider_d(cand, img, refs_all=REFS, N=4, sigma=6.0):
    """Simplified CIDEr-D: TF-IDF n-gram vectors (IDF over the images' reference sets),
    clipped cosine with each reference, Gaussian length penalty, x10, mean over n."""
    n_img = len(refs_all)
    def vec(words, n):
        df = lambda g: sum(any(g in ngrams(r.split(), n) for r in rs) for rs in refs_all)
        return {g: k * np.log(n_img / max(1, df(g))) for g, k in ngrams(words, n).items()}
    c = cand.split()
    score = 0.0
    for n in range(1, N + 1):
        gc = vec(c, n)
        nc = np.sqrt(sum(v * v for v in gc.values()))
        s_n = 0.0
        for ref in refs_all[img]:
            r = ref.split()
            gr = vec(r, n)
            nr = np.sqrt(sum(v * v for v in gr.values()))
            dot = sum(min(v, gr.get(g, 0.0)) * gr.get(g, 0.0) for g, v in gc.items())
            if nc > 0 and nr > 0:
                s_n += np.exp(-(len(c) - len(r)) ** 2 / (2 * sigma ** 2)) * dot / (nc * nr)
        score += 10 * s_n / len(refs_all[img]) / N
    return score

candidates = {
    "specific":     "a red circle left of a blue square",
    "generic":      "two shapes on a black background",
    "vague":        "a circle and a square",
    "hallucinated": "a red circle left of a blue square and a green triangle",
    "wrong object": "a red circle left of a blue triangle",
    "swapped":      "a blue circle left of a red square",
}
print(f"{'candidate':13s} {'BLEU-1':>6s} {'BLEU-2':>6s} {'BLEU-4':>6s} {'CIDEr-D':>7s}")
for name, cand in candidates.items():
    b = [bleu(cand, REFS[0], N) for N in (1, 2, 4)]
    print(f"{name:13s} {b[0]:6.3f} {b[1]:6.3f} {b[2]:6.3f} {cider_d(cand, 0):7.3f}")

# IDF over the six images: words used for every image carry no weight
idf = {w: np.log(6 / sum(any(w in r.split() for r in rs) for rs in REFS))
       for w in ["a", "of", "circle", "red", "triangle"]}
print("IDF:", ", ".join(f"{w} {v:.2f}" for w, v in idf.items()))

candidate     BLEU-1 BLEU-2 BLEU-4 CIDEr-D
specific       1.000  1.000  0.946   5.319
generic        0.141  0.000  0.000   0.000
vague          0.670  0.474  0.000   1.229
hallucinated   0.750  0.739  0.618   4.011
wrong object   0.875  0.866  0.783   3.742
swapped        1.000  0.845  0.517   2.695
IDF: a 0.00, of 0.69, circle 0.41, red 0.69, triangle 0.69

The specific caption wins. The generic one scores zero CIDEr-D: its only word in scene 0’s references is “a”, whose IDF is zero. The vague caption is true but scores far below the hallucinated one, which invents a green triangle yet keeps CIDEr-D 4.0 and BLEU-4 0.62. The swapped caption gets a perfect BLEU-1; only longer n-grams notice the misbound colors. Sentence-level BLEU-4 is zero whenever no 4-gram matches, one reason BLEU is computed over a corpus.

Applications

• Accessibility: alt text and scene descriptions for blind and low-vision users, a motivation named in Show and Tell.
• Search and indexing: captions make image collections searchable by text.
• Training data: synthetic captions filter or replace noisy web alt-text for pretraining, as in BLIP’s CapFilt.

Open Problems

• Object hallucination. Rohrbach et al. (2018) found that 7.4% to 17.7% of Karpathy-test captions from the models they tested mentioned an absent object, that attention over detected regions hallucinated less, and that SCST’s CIDEr optimization often increased hallucination, which they suggest is because CIDEr does not penalize it enough. Multimodal Hallucination covers the problem in vision–language models.
• Generic captions. By our reasoning, maximum-likelihood training and beam search favor frequent phrasings that fit many images; CIDEr’s IDF weighting discounts such n-grams as less informative (Vedantam et al., 2015).
• Fine-grained detail. SPICE’s breakdown makes the gap visible: for Up-Down after CIDEr optimization, the F-score was 39.1 for objects but 10.0 for attributes and 6.5 for relations (Anderson et al., 2018).
• Evaluating long descriptions. In our view, the long descriptions of vision–language models fit poorly with n-gram metrics built for one-sentence references, and CLIP-based scores avoid references but inherit the scoring model’s blind spots.