CIDEr

CIDEr (Consensus-based Image Description Evaluation) scores a generated caption by how closely it matches the way most people describe the image. Every caption is turned into vectors of n-gram weights, with $n = 1, \dots, 4$, in which an n-gram counts for more when it is frequent in the sentence and rare across the images being evaluated (TF-IDF weighting). The score is the average cosine similarity between the candidate’s vectors and those of each human reference. Vedantam, Zitnick and Parikh (2015) proposed it together with a human protocol for measuring consensus and two datasets with 50 reference sentences per image, PASCAL-50S and ABSTRACT-50S. A modified version, CIDEr-D, adds count clipping and a sentence-length penalty and is the “CIDEr” of the COCO caption evaluation code. Image Captioning defines both versions in brief, and BLEU, the precision-based metric CIDEr was designed against, is the prerequisite here. This article gives the paper’s definition and evidence, the official code’s departures from it, and an example that holds one set of captions fixed and changes what the score depends on.

Formula

For image $I_i$ with candidate $c_i$ and references $S_i = \{s_{i1}, \dots, s_{im}\}$, let $h_k(s)$ be the number of times n-gram $\omega_k$ occurs in sentence $s$. The paper first maps every word to its stem (“fishes”, “fishing” and “fished” become “fish”). Its Eq. 1 weights each n-gram by term frequency times inverse document frequency:

$$g_k(s_{ij}) = \frac{h_k(s_{ij})}{\sum_{\omega_l \in \Omega} h_l(s_{ij})} \, \log\!\left( \frac{|I|}{\sum_{I_p \in I} \min\big(1, \sum_q h_k(s_{pq})\big)} \right),$$

where $\Omega$ is the vocabulary of all n-grams and $I$ the set of all images in the dataset. The denominator inside the logarithm is the number of images in whose reference sentences $\omega_k$ occurs at least once, so an n-gram used for every image gets weight 0. Writing $g^n(s)$ for the vector of weights of all n-grams of length $n$, the per-order score (Eq. 2) and the combined score (Eq. 3) are

$$\text{CIDEr}_n(c_i, S_i) = \frac{1}{m} \sum_{j} \frac{g^n(c_i) \cdot g^n(s_{ij})}{\lVert g^n(c_i) \rVert \, \lVert g^n(s_{ij}) \rVert}, \qquad \text{CIDEr}(c_i, S_i) = \sum_{n=1}^{N} w_n \, \text{CIDEr}_n(c_i, S_i),$$

with $N = 4$ and uniform weights $w_n = 1/N$, which the authors found empirically to work best. In the paper’s account, the averaged cosine covers both precision and recall.

Two details of Eq. 1 are left open. The paper does not say whether the term-frequency normalizer $\sum_{\omega_l \in \Omega} h_l$ runs over the n-grams of every length or only those of length $n$. In Eqs. 2 and 3 the choice does not matter, because a scalar factor on a whole vector cancels in a cosine. And an n-gram of the candidate that occurs in no reference has a document frequency of 0, so its IDF is undefined. The example below computes both readings: a floor of 1 on the document frequency, as the official code does, and the limit of an ever larger weight, under which that order’s cosine goes to 0.

CIDEr-D. Section 8 of the paper, on gameability, proposes three changes as a defence against future gaming of the metric. Stemming is removed, so that the correct forms of words must be used. A Gaussian penalty on the difference between the candidate’s and the reference’s lengths $l(\cdot)$ counters the higher scores that, in some cases, basic CIDEr gives to long sentences that repeat confident words. Clipping counters the gaming of that penalty by repetition up to the desired length. The result is Eq. 4, combined over $n$ by Eq. 5 with uniform weights:

$$\text{CIDEr-D}_n(c_i, S_i) = \frac{10}{m} \sum_{j} e^{-\frac{(l(c_i) - l(s_{ij}))^2}{2\sigma^2}} \, \frac{\min\big(g^n(c_i), g^n(s_{ij})\big) \cdot g^n(s_{ij})}{\lVert g^n(c_i) \rVert \, \lVert g^n(s_{ij}) \rVert}, \qquad \text{CIDEr-D} = \sum_{n=1}^{N} w_n \, \text{CIDEr-D}_n,$$

with $\sigma = 6$. The factor 10, the authors write, makes CIDEr-D numerically similar to other metrics. They report a Spearman’s ρ of 0.94 between CIDEr-D and the original CIDEr.

The paper disagrees with itself on clipping. The text of §8 says that for each n-gram the number of candidate occurrences is clipped to the number of reference occurrences. Eq. 4 instead takes the elementwise minimum of the weight vectors $g^n$, whose term frequencies are normalized by each sentence’s length. The two agree only when candidate and reference have the same normalizer. Writing $L$ for that normalizer, the clipped term of Eq. 4 equals the count-clipped term with the reference count $h_k(s)$ replaced by $h_k(s)\,L(c)/L(s)$, so by our derivation Eq. 4 gives at least the count-clipped value when the candidate is the longer sentence and at most that value when it is the shorter. The official code clips counts, as the text says. The example checks the derivation under both readings of $L$.

Range. Every cosine lies between 0 and 1, because the weights are not negative when the IDF is counted over the evaluated images. Paper CIDEr therefore lies in $[0, 1]$, and CIDEr-D, with its factor 10, in $[0, 10]$. The official scorers multiply both CIDEr and CIDEr-D by 10. Results tables then print the scorer’s value multiplied by 100, as the tables in Image Captioning and Video Captioning do, so on that convention a COCO captioning CIDEr of 127.8, the kind of number BLIP reports, corresponds to a scorer output of 1.278.

Intuition

The paper’s starting point is a distinction between two goals for a caption. In their words, “What humans like often does not correspond to what is human-like”. They chose to measure the second: how similar a sentence is to the majority, or consensus, of the descriptions people write for the image. A consensus measure, they argue, should reward n-grams of the candidate that occur in the references, penalize n-grams that do not, and give less weight to n-grams common across all images, since those are likely to be less informative. TF-IDF weighting, borrowed from text retrieval (the same weighting that bag of visual words applies to visual words), does all three: the term frequency rewards n-grams the references use often, and the IDF discounts words such as “a” or “the” that describe every image. Longer n-grams, the authors write, capture grammatical properties and richer semantics.

Two design choices separate CIDEr from BLEU. BLEU takes, for each n-gram, the maximum count over the references; the paper explains BLEU’s loss of accuracy with more references by this maximum, which makes a match with a lower-quality reference more likely. CIDEr averages over all references, so in our reading every reference pulls the candidate towards the common description. And because a cosine compares the shape of two vectors, CIDEr rewards a candidate for using the references’ n-grams in about their proportions, which covers both precision and recall, where BLEU measures precision and adds a brevity penalty.

Interpretation

A CIDEr score is relative to two things outside the caption: the reference set of the image and the set of images evaluated together. The IDF counts documents over the reference sets of the evaluated images, so the same caption with the same references scores differently in another evaluation set. Video Captioning and Flickr30k note the consequence for benchmarks: a model scored on a subset of a test set is scored with different IDF weights. A single image evaluated alone gets exactly 0, because every n-gram of its references occurs in 1 of 1 images and has IDF $\log 1 = 0$; the official README warns of this. Scores are therefore comparable only on the same split with the same references, and only between scorers with the same IDF source.

The scale also needs care. Paper CIDEr lies between 0 and 1. The code’s CIDEr and CIDEr-D lie between 0 and 10. Papers multiply by 100. For scale, Image Captioning reports that one human caption scored against the other COCO references of its image gets CIDEr-D 85.4 (on the ×100 scale), and the models in its results table exceed it. In our reading, a score above the human one means that the model reproduces the references’ common phrasing more closely than a single person does, which is what a consensus metric measures.

The consensus protocol. To evaluate metrics against people, the authors built a human measure of consensus. Subjects saw three sentences A, B and C and were asked which of B and C was more similar to A, without being told what “similar” means. B and C are two candidates, and A runs over all references of the image, one triplet per reference. Each triplet got three judgments and a majority vote, and B won the pair if it was chosen by a majority of triplets. A metric’s accuracy is the fraction of candidate pairs on which it picks the same winner as people. The authors relate the relative question to two-alternative forced choice in psychophysics, and write that it is easier than an absolute rating from 1 to 5. All studies ran on Amazon Mechanical Turk with workers in the United States.

PASCAL-50S and ABSTRACT-50S. Existing datasets had at most five descriptions per image, which the authors found too few to measure what a majority of people would say. PASCAL-50S has 50 sentences for each of the 1,000 images of the UIUC Pascal Sentence Dataset, and ABSTRACT-50S 50 sentences for 500 random images of Zitnick and Parikh’s Abstract Scenes Dataset of clipart scenes. Subjects were asked to describe what was going on in the image, capturing its main aspects in a way others were also likely to write; 683 subjects wrote for PASCAL-50S and 465 for ABSTRACT-50S, each sentence of an image by a different subject. Average sentence lengths are 8.8 words for PASCAL-50S and 10.59 for ABSTRACT-50S. The paper also reports that most metrics improved with more sentences, and that, inspired by this finding, the MS COCO testing set has 5,000 images with 40 references each. It leaves qualitative examples, other versions of each metric and snapshots of its interfaces to an arXiv version (Vedantam et al., 2014), which we did not read.

Candidate pairs. On ABSTRACT-50S, 48 sentences per image are references and the remaining two serve as candidates, in 400 pairs: 200 human–human correct (HC; two descriptions of the same image) and 200 human–human incorrect (HI; one description of another image). PASCAL-50S has 4,000 pairs, 1,000 each of HC, HI, human–machine (HM) and machine–machine (MM), with machine sentences from five systems (Midge, Babytalk, Story, and the Video and Video+ versions of Rohrbach et al.’s video-to-text method). Two references are dropped on PASCAL-50S so that both datasets use at most 48. The authors implemented the other metrics themselves, computing BLEU at sentence level, and used METEOR’s version 1.5 code, also aggregated per sentence.

When to Use It

The paper’s evidence is about one use: predicting which of two sentences people find closer to the consensus of an image’s descriptions, given many references.

• How many references. In a pilot on 50 Pascal images with 120 sentences each (179 pairs of all four kinds), the text reports that accuracy improved for all three metrics tested (BLEU$_1$, ROUGE$_1$ and CIDEr$_1$) over the first 10 references. ROUGE$_1$ rose from 0.63 with one reference to 0.77 with five, and ROUGE$_1$ and CIDEr$_1$ kept improving up to about 50 references, after which gains were small. BLEU$_1$ got worse with more references; the authors explain that BLEU’s maximum over references makes matching a lower-quality reference more likely as references are added. The datasets were then collected with 50 sentences per image.
• Accuracy against other metrics. With 48 references, CIDEr was the most accurate metric on both datasets, at 84% on each, followed by METEOR (Fig. 3). With five references, CIDEr beat METEOR on ABSTRACT-50S, and METEOR was better on PASCAL-50S, which the authors attribute to METEOR’s soft similarity helping with few references. On PASCAL-50S, BLEU$_1$ and ROUGE$_1$ at five references reached 76% and 74%.
• By pair type (Table 1, best version of each metric). On PASCAL-50S, CIDEr scored 71.8 (HC), 99.7 (HI), 92.1 (HM) and 72.2 (MM), against BLEU$_4$’s 64.8, 97.7, 93.8 and 63.6 and METEOR’s 65.2, 99.3, 96.4 and 67.7. On ABSTRACT-50S, CIDEr scored 71.5 (HC) and 96.0 (HI), tied with ROUGE on HC. CIDEr was best in five of the six columns; METEOR was best on HM. The ROUGE row is ROUGE$_L$ on PASCAL-50S and ROUGE$_1$ on ABSTRACT-50S. The authors point to the gains on HC and MM, the pairs that differ in fine detail, and note that all metrics do well on HI and HM.
• Against human performance. With ground-truth consensus from 24 references and the other 24 used for prediction by both people and metrics, CIDEr reached 82% on PASCAL-50S against 90% for people, and 82% on ABSTRACT-50S against 83%.
• Ranking systems. On 100 PASCAL images with all ten pairs of the five systems per image (1,000 pairs, each tripled with 20 random references), people and CIDEr both ranked Midge first, Babytalk second and Story last, and disagreed on the two video systems. The Pearson correlation between the systems’ win fractions under people and under CIDEr was 0.98.
• A rating dataset. On Flickr8K, whose human judgments the paper describes as 1–5 ratings, the paper reports a Spearman’s ρ of 0.58 for CIDEr, credited in a footnote to Desmond Elliott, against 0.56 for METEOR, citing Elliott and Keller (2014). The cited paper differs on both points: its Table 1 gives METEOR’s sentence-level ρ on Flickr8K as 0.524 (17,466 judgments, METEOR 1.4.0), and it describes the Flickr8K judgments as expert ratings on a scale of 1–4. The CIDEr paper does not explain the difference.

The paper made CIDEr-D available on the MS COCO caption evaluation server, and the coco-caption scorer named CIDEr computes CIDEr-D (read in the code). In Image Captioning, self-critical sequence training uses it as the reward. CIDEr suits evaluation sets with many references per image and many images, where the IDF has something to count. It suits a single image badly, and comparisons across papers only when they share the split, the references and the scorer.

Failure Cases

• Optimizing the metric. The paper’s §8 warns that optimizing an algorithm for a specific metric can give sentences with high scores that people judge poor, and CIDEr-D’s changes are its defence. Image Captioning reports that SCST’s CIDEr optimization often increased object hallucination in the models Rohrbach et al. (2018) tested, which they suggest is because CIDEr does not penalize it enough.
• Evaluation-set dependence. A caption’s score changes when other images are added to or removed from the evaluation set, even if its own references stay the same; the example measures this for every image. One image alone scores 0.
• Consensus is not correctness. CIDEr rewards the references’ common phrasing. A true but unusual description, or one with synonyms the references do not use, scores low, since there is no soft matching of the kind METEOR has. A wrong word that happens to be frequent in the references can score well, as Image Captioning’s example of a hallucinated but well-phrased caption shows.
• Few references. The paper’s own experiments show that consensus metrics need many references. With five, METEOR was more accurate on PASCAL-50S, and Video Captioning notes that S2VT chose METEOR for single-reference movie datasets for that reason.
• Different scorers under one name. The coco-caption scorer named “CIDEr” computes CIDEr-D with corpus IDF, while the official CIDEr repository offers both CIDEr and CIDEr-D, and its demo configuration (params.json) takes the IDF from the COCO validation set. Without the scorer and the IDF source, two “CIDEr” numbers may not be the same quantity.

Variants

The official code

Two repositories carry the authors’ code, both under a BSD 2-clause licence: ramavedantam/cider (formerly vrama91/cider), read at commit f281464, and the coco-caption package, read at commit 3a9afb2. They differ from the paper and from each other in these ways:

	Paper (Eqs. 1–5)	ramavedantam/cider “CIDEr”	ramavedantam/cider “CIDEr-D”	coco-caption “CIDEr”
Stemming	CIDEr yes, CIDEr-D no	no	no	no
Term frequency	count / sentence length	raw count	raw count	raw count
IDF of unseen n-gram	undefined	document frequency floored at 1	floored at 1	floored at 1
Clipping	CIDEr-D: on weight vectors (Eq. 4); counts in the text	none	counts	counts
Length penalty	CIDEr-D, $\sigma = 6$	none	$\sigma = 6$	$\sigma = 6$
Factor 10	CIDEr-D only	yes	yes	yes
IDF source	images in the dataset	corpus or a precomputed file	same	evaluated images only

Details, read in the source:

• Raw counts. Both scorers weight each n-gram by its raw count times $\log|I| - \log\max(1, \text{df})$. Since a cosine ignores a scalar factor on a vector, the missing length normalization does not change unclipped CIDEr, and by our reading the code’s CIDEr is 10 times the paper’s Eq. 3 without stemming, provided the paper’s undefined IDF is read as the code’s floor. The example checks this, and shows it fails for the other reading.
• Clipping. CIDEr-D computes min(vec_hyp, vec_ref) * vec_ref on the weights. Both weights of an n-gram share its IDF, so this is count clipping, as the paper’s text describes.
• Length. The length passed to the penalty counts the sentence’s bigrams, not its words. For a non-empty sentence the bigram count is the word count minus one, so the difference of lengths is the same as for word counts.
• IDF source. The README warns that with the default corpus mode a reference set of one image scores 0, and recommends coco-val-df for one or a few images. That mode uses $\log 40504$ (the size of COCO’s validation set) and document frequencies from a pickled file in the repository’s data folder, and the demo configuration params.json selects it. The docstring of cider.py names the accepted values as corpus and coco-train. coco-caption has no such option: its IDF always comes from the evaluated images.
• Sigma. coco-caption’s wrapper passes $\sigma$ to the scorer. In ramavedantam’s ciderD.py, the wrapper accepts a sigma argument but constructs the scorer with only the n-gram order, so the scorer’s default of 6.0 is used.
• Averaging. Each image’s score is the mean over $n$ of the per-order sums, divided by the number of references and multiplied by 10; the corpus score is the mean over images.
• Tokenization. Captions are tokenized by the Stanford CoreNLP 3.4.1 PTB tokenizer with lowercasing, and a fixed list of punctuation tokens is removed, before scoring. Video Captioning describes how captioning papers use this pipeline.

Example: what CIDEr’s numbers depend on

The script renders 300 synthetic 64×64 scenes. Each holds one to four copies of a single shape (circle, square or triangle) in a single colour, one copy per quadrant so that objects never overlap; the colours rotate so that each appears in 25 of the first 100 scenes. Those 100 are the evaluation set. The other 200 form a separate pool whose reference document frequencies play the role of the code’s precomputed coco-val-df file, in our analogue with $|I| = 200$. Each image has ten reference captions from templates such as “there are three red circles” and “three circles coloured red”, in lower case without punctuation, so whitespace splitting gives the same tokens the PTB tokenizer would; the order of an image’s references is a seeded permutation, and the 1-, 2-, 3- and 5-reference sets are its first $k$.

One rule-based captioner reads the pixels: it thresholds each colour, counts connected components with scikit-image and names the shape from how much of its bounding box it fills. It writes “three red circle”, with the noun always singular, so every image with two or more objects carries a known agreement error, the kind of error the paper’s stemming forgives and CIDEr-D does not.

The captions stay fixed. The script scores them with:

• the official code’s CIDEr and CIDEr-D, with their CIDEr-specific functions re-typed in Python 3 from ramavedantam/cider at f281464 (our own n-gram counter replaces the original’s);
• the paper’s Eqs. 1–3 by hand, under both readings of the undefined IDF, and with and without a toy stemmer that maps plural nouns to their singular (our analogue; the paper does not name its stemmer);
• the paper’s Eqs. 4–5 by hand, under both readings of the length normalizer;
• an independent form of the code’s CIDEr-D written with clipped counts.

It then prints the checks and hypotheses planned before the first run:

• C1: the code’s CIDEr equals 10 times the paper’s (floor reading, no stemming) to within $10^{-12}$, for each caption and for the caption repeated two to four times; and the code’s CIDEr-D equals the count-clipped form to within $10^{-12}$.
• C2: the derived ordering between Eq. 4 and the code’s clipped terms holds for every image, reference and order, under both readings.
• C3: an image scored alone gets exactly 0 with corpus IDF, and never 0 with the external IDF.
• H4: each image’s CIDEr-D differs between two 25-image evaluation sets that contain it: all 25 images of its colour, or the image and 8 images of each other colour. The direction was not predicted. H4 was written after a planning script had shown such a difference for one caption on other data.
• H5: each caption repeated $k = 2, 3, 4$ times keeps its unclipped unigram term exactly (H5a, derived: counts scale by $k$ and the IDF depends only on the references); its clipped unigram term without the penalty falls to exactly $1/k$ (H5b, derived: each candidate word occurs once in the caption and, as the data check prints, at most once in any reference); and its full CIDEr-D falls below the unrepeated value on every image (H5c). H5c was written after a planning script had shown CIDEr-D cut by repetition for one caption on other data. The tolerances of C1–C3 were set from a planning script that ran the same set-up and printed only the checked quantities.

The number of references (R) and stemming (S) are printed without hypotheses. The script runs in about 2–3 seconds (Python 3.13.9, NumPy 2.3.5, scikit-image 0.25.2) and prints the same output on every run.

"""What CIDEr's numbers depend on: one rule-based captioner reads 100 synthetic shape scenes, and its
fixed captions are scored by the paper's CIDEr (Eqs. 1-3), the paper's CIDEr-D (Eqs. 4-5) and the
official code's CIDEr and CIDEr-D, under the choices the score depends on.
NumPy 2.3.5, scikit-image 0.25.2."""
import os
os.environ["OMP_NUM_THREADS"] = "1"; os.environ["OPENBLAS_NUM_THREADS"] = "1"
from collections import Counter, defaultdict
import numpy as np
from skimage.draw import disk, polygon, rectangle
from skimage.measure import label, regionprops

COLOURS = {"red": (1, 0, 0), "green": (0, 1, 0), "blue": (0, 0, 1), "yellow": (1, 1, 0)}
SHAPES = ["circle", "square", "triangle"]
NUMBER = {1: "one", 2: "two", 3: "three", 4: "four"}
CELLS = [(16, 16), (16, 48), (48, 16), (48, 48)]
N_IMG, N_POOL, SIZE, SEED = 100, 200, 64, 0

# ---------------------------------------------------------------- scenes, references, captioner
def make_scene(rng, colour):
    shape, count = SHAPES[rng.integers(3)], int(rng.integers(1, 5))
    img = np.zeros((SIZE, SIZE, 3))
    for k in rng.permutation(4)[:count]:                      # one object per grid cell: no overlaps
        r, c = CELLS[k][0] + rng.integers(-3, 4), CELLS[k][1] + rng.integers(-3, 4)
        s = int(rng.integers(6, 10))
        if shape == "circle":
            rr, cc = disk((r, c), s + 0.5, shape=img.shape[:2])
        elif shape == "square":
            rr, cc = rectangle((r - s, c - s), (r + s, c + s), shape=img.shape[:2])
        else:
            rr, cc = polygon([r - s, r + s, r + s], [c, c - s, c + s], shape=img.shape[:2])
        img[rr, cc] = COLOURS[colour]
    return img, dict(count=count, colour=colour, shape=shape)

def references(sc):
    n, col, shp = sc["count"], sc["colour"], sc["shape"]
    N, noun, verb = NUMBER[n], shp if n == 1 else shp + "s", "is" if n == 1 else "are"
    A = f"{N} {col} {noun}"
    return [A, f"there {verb} {A}", f"{A} on a black background", f"a black picture with {A}",
            f"{N} {noun} coloured {col}", f"the image shows {A}", f"{N} {col} {shp} shapes",
            f"a single {col} {noun}" if n == 1 else f"some {col} {noun}",
            f"{A} spread over the image", f"a lone {col} {noun}" if n == 1 else f"a group of {N} {col} {noun}"]

def caption(img):
    """Threshold each colour, count connected components, name the shape from the bounding-box fill
    (extent): square about 1, disc about 0.8, triangle about 0.5. Writes the noun in the singular."""
    for name, rgb in COLOURS.items():
        m = np.all(np.isclose(img, rgb), axis=2)
        props = regionprops(label(m))
        if props:
            e = np.mean([p.extent for p in props])
            shape = "square" if e > 0.9 else "circle" if e > 0.65 else "triangle"
            return f"{NUMBER[len(props)]} {name} {shape}", (len(props), name, shape)

def ngrams(sentence, n=4):
    w = sentence.split()
    return Counter(tuple(w[i:i + k]) for k in range(1, n + 1) for i in range(len(w) - k + 1))

# ---------------------------------------------------------------- official code, re-typed
# Re-typed in Python 3 from ramavedantam/cider at commit f281464 (pyciderevalcap/cider/cider_scorer.py
# and pyciderevalcap/ciderD/ciderD_scorer.py: CiderScorer.compute_doc_freq, compute_cider and
# compute_score). The n-gram counter above is our own. Licence of the original:
#
# Copyright (c) 2015, Xinlei Chen, Hao Fang, Tsung-Yi Lin, and Ramakrishna Vedantam
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#
# 1. Redistributions of source code must retain the above copyright notice, this
#    list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright notice,
#    this list of conditions and the following disclaimer in the documentation
#    and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
# The views and conclusions contained in the software and documentation are those
# of the authors and should not be interpreted as representing official policies,
# either expressed or implied, of the FreeBSD Project.
def code_cider(cands, refs, variant="CIDEr-D", df_mode="corpus", ext=None, n=4, sigma=6.0):
    """variant 'CIDEr' (cider_scorer.py) or 'CIDEr-D' (ciderD_scorer.py; coco-caption's "CIDEr").
    df_mode 'corpus': IDF over the evaluated images' references; 'external': ext = (df, |I|), the role
    the code's precomputed 'coco-val-df' file plays. Returns (mean, per-image scores, per-image per-n)."""
    crefs = [[ngrams(r, n) for r in rs] for rs in refs]
    ctest = [ngrams(c, n) for c in cands]
    if df_mode == "corpus":
        document_frequency = defaultdict(float)
        for rs in crefs:
            for ngram in set(g for r in rs for g in r):
                document_frequency[ngram] += 1
        assert len(ctest) >= max(document_frequency.values())
        ref_len = np.log(float(len(crefs)))
    else:
        document_frequency = defaultdict(float, ext[0])
        ref_len = np.log(float(ext[1]))

    def counts2vec(cnts):
        vec = [defaultdict(float) for _ in range(n)]
        length, norm = 0, [0.0 for _ in range(n)]
        for ngram, term_freq in cnts.items():
            df = np.log(max(1.0, document_frequency[ngram]))     # unseen n-gram: df floored at 1
            k = len(ngram) - 1
            vec[k][ngram] = float(term_freq) * (ref_len - df)     # raw count x IDF, no length-normalized TF
            norm[k] += pow(vec[k][ngram], 2)
            if k == 1:
                length += term_freq                               # "length" counts bigrams
        return vec, [np.sqrt(x) for x in norm], length

    def sim(vec_hyp, vec_ref, norm_hyp, norm_ref, length_hyp, length_ref):
        delta = float(length_hyp - length_ref)
        val = np.array([0.0 for _ in range(n)])
        for k in range(n):
            for ngram, count in vec_hyp[k].items():
                if variant == "CIDEr-D":                          # clipping
                    val[k] += min(vec_hyp[k][ngram], vec_ref[k][ngram]) * vec_ref[k][ngram]
                else:
                    val[k] += vec_hyp[k][ngram] * vec_ref[k][ngram]
            if (norm_hyp[k] != 0) and (norm_ref[k] != 0):
                val[k] /= (norm_hyp[k] * norm_ref[k])
            assert not np.isnan(val[k])
            if variant == "CIDEr-D":                              # Gaussian length penalty
                val[k] *= np.e ** (-(delta ** 2) / (2 * sigma ** 2))
        return val

    scores, per_n = [], []
    for test, rs in zip(ctest, crefs):
        vec, norm, length = counts2vec(test)
        score = np.array([0.0 for _ in range(n)])
        for ref in rs:
            vec_ref, norm_ref, length_ref = counts2vec(ref)
            score += sim(vec, vec_ref, norm, norm_ref, length, length_ref)
        per_n.append(score / len(rs) * 10.0)
        score_avg = np.mean(score)                                # mean over n
        score_avg /= len(rs)                                      # mean over references
        score_avg *= 10.0                                         # factor 10
        scores.append(score_avg)
    return np.mean(np.array(scores)), np.array(scores), np.array(per_n)
# ---------------------------------------------------------------- end of re-typed code

# ---------------------------------------------------------------- the paper's equations, by hand
STEM = {"circles": "circle", "squares": "square", "triangles": "triangle", "shapes": "shape"}

def stem(s):                     # our toy stemmer: the paper does not name one; plurals only
    return " ".join(STEM.get(w, w) for w in s.split())

def doc_freq(refs):
    df = Counter()
    for rs in refs:
        df.update(set(g for r in rs for g in ngrams(r)))
    return df

def paper_g(s, df, n_images, unseen):
    """Eq. 1: g_k(s) = h_k(s) / sum_l h_l(s) * log(|I| / df_k), with the TF normalizer over all n-grams
    of s. An n-gram in no reference has df = 0: 'floor' uses df = 1 (as the code), 'limit' returns inf."""
    h = ngrams(s); total = sum(h.values())
    g = {}
    for k, c in h.items():
        if df[k] == 0:
            g[k] = np.inf if unseen == "limit" else c / total * np.log(n_images)
        else:
            g[k] = c / total * np.log(n_images / df[k])
    return g

def cos_n(gc, gs, order):
    a = {k: v for k, v in gc.items() if len(k) == order}
    b = {k: v for k, v in gs.items() if len(k) == order}
    if any(np.isinf(v) for v in a.values()):
        return 0.0               # infinite weight on an n-gram no reference has: the cosine tends to 0
    na, nb = np.sqrt(sum(v * v for v in a.values())), np.sqrt(sum(v * v for v in b.values()))
    if na == 0 or nb == 0:
        return 0.0
    return sum(v * b.get(k, 0.0) for k, v in a.items()) / (na * nb)

def paper_cider(cands, refs, stemmed=False, unseen="floor"):
    """Eqs. 2-3: mean cosine over references per n, uniform weights 1/4. No factor 10."""
    if stemmed:
        cands, refs = [stem(c) for c in cands], [[stem(r) for r in rs] for rs in refs]
    df, I = doc_freq(refs), len(refs)
    out = []
    for c, rs in zip(cands, refs):
        gc = paper_g(c, df, I, unseen)
        out.append(np.mean([np.mean([cos_n(gc, paper_g(r, df, I, unseen), o) for r in rs])
                            for o in range(1, 5)]))
    return np.array(out)

def paper_cider_d(cands, refs, omega="all", sigma=6.0):
    """Eqs. 4-5: clip min(g^n(c), g^n(s)) on the length-normalized TF-IDF weights, Gaussian on word
    counts, factor 10. omega 'all': TF normalizer over n-grams of every order; 'order': over order n."""
    df, I = doc_freq(refs), len(refs)
    def g(s, order):
        h = {k: v for k, v in ngrams(s).items() if len(k) == order}
        total = sum(ngrams(s).values()) if omega == "all" else sum(h.values())
        return {k: v / total * np.log(I / max(1, df[k])) for k, v in h.items()}
    out = []
    for c, rs in zip(cands, refs):
        terms = []
        for o in range(1, 5):
            gc, acc = g(c, o), 0.0
            for r in rs:
                gs = g(r, o)
                nc, ns = np.sqrt(sum(v * v for v in gc.values())), np.sqrt(sum(v * v for v in gs.values()))
                num = sum(min(v, gs.get(k, 0.0)) * gs.get(k, 0.0) for k, v in gc.items())
                pen = np.exp(-(len(c.split()) - len(r.split())) ** 2 / (2 * sigma ** 2))
                acc += pen * num / (nc * ns) if nc > 0 and ns > 0 else 0.0
            terms.append(10 * acc / len(rs))
        out.append(np.mean(terms))
    return np.array(out)

def count_clip(cands, refs, gaussian=True, sigma=6.0):
    """Independent form of the code's CIDEr-D: clip raw counts, min(h_c, h_s) * h_s * idf^2."""
    df, I = doc_freq(refs), len(refs)
    idf = lambda k: np.log(I) - np.log(max(1, df[k]))
    out, uni = [], []
    for c, rs in zip(cands, refs):
        hc, terms = ngrams(c), np.zeros(4)
        for r in rs:
            hs = ngrams(r)
            pen = np.exp(-(len(c.split()) - len(r.split())) ** 2 / (2 * sigma ** 2)) if gaussian else 1.0
            for o in range(1, 5):
                a = {k: v for k, v in hc.items() if len(k) == o}
                b = {k: v for k, v in hs.items() if len(k) == o}
                na = np.sqrt(sum((v * idf(k)) ** 2 for k, v in a.items()))
                nb = np.sqrt(sum((v * idf(k)) ** 2 for k, v in b.items()))
                if na > 0 and nb > 0:
                    terms[o - 1] += pen * sum(min(v, b.get(k, 0)) * b.get(k, 0) * idf(k) ** 2
                                              for k, v in a.items()) / (na * nb)
        out.append(10 * terms.mean() / len(rs)); uni.append(10 * terms[0] / len(rs))
    return np.array(out), np.array(uni)

# ---------------------------------------------------------------- data
rng = np.random.default_rng(SEED)
scenes, imgs = [], []
for i in range(N_IMG + N_POOL):                     # first 100: evaluated; last 200: external-IDF pool
    img, sc = make_scene(rng, list(COLOURS)[i % 4]); imgs.append(img); scenes.append(sc)
REFS_ALL = []
for sc in scenes:
    r = references(sc); REFS_ALL.append([r[j] for j in rng.permutation(len(r))])
CANDS, READ = zip(*[caption(im) for im in imgs[:N_IMG]])
CANDS = list(CANDS)
REFS = REFS_ALL[:N_IMG]
POOL_DF = doc_freq(REFS_ALL[N_IMG:])

print("data check")
print(f"  images {N_IMG}, pool {N_POOL}, references per image {len(REFS[0])}")
print(f"  images per colour {[sum(s['colour'] == c for s in scenes[:N_IMG]) for c in COLOURS]}, "
      f"per count {[sum(s['count'] == k for s in scenes[:N_IMG]) for k in NUMBER]}")
acc = [READ[i] == (scenes[i]["count"], scenes[i]["colour"], scenes[i]["shape"]) for i in range(N_IMG)]
print(f"  captioner reads count, colour and shape correctly on {sum(acc)}/{N_IMG} images")
print(f"  fewest words in a caption or reference {min(len(s.split()) for s in CANDS + sum(REFS, []))}")
mx = max(r.split().count(w) for c, rs in zip(CANDS, REFS) for r in rs for w in c.split())
print(f"  most times a candidate word occurs in one reference {mx}")
print(f"  example: candidate '{CANDS[2]}'; references '{REFS[2][0]}', '{REFS[2][1]}'")

# ---------------------------------------------------------------- C1: code against paper
print("\nC1  code against the paper (100 images, 10 references, corpus IDF)")
code_c = code_cider(CANDS, REFS, "CIDEr")[1]
code_d = code_cider(CANDS, REFS, "CIDEr-D")[1]
DF_REFS = doc_freq(REFS)
for k in (1, 2, 3, 4):
    rep = [" ".join([c] * k) for c in CANDS]
    p_floor, p_limit = paper_cider(rep, REFS, unseen="floor"), paper_cider(rep, REFS, unseen="limit")
    cc = code_cider(rep, REFS, "CIDEr")[1]
    unseen = sum(any(DF_REFS[g] == 0 for g in ngrams(c)) for c in rep)
    print(f"  x{k}: max |code CIDEr - 10 x paper (floor)| {np.max(np.abs(cc - 10 * p_floor)):.1e}; "
          f"captions with an n-gram in no reference {unseen:3d}; mean 10 x paper floor {10 * p_floor.mean():.4f}, "
          f"limit {10 * p_limit.mean():.4f}")
cd_ind = count_clip(CANDS, REFS)[0]
print(f"  max |code CIDEr-D - count-clipped form| {np.max(np.abs(code_d - cd_ind)):.1e}")

# ---------------------------------------------------------------- C2: the paper's clipped numerator
print("\nC2  paper Eq. 4 against code CIDEr-D, per (image, reference, n) term before the penalty")
df, I = doc_freq(REFS), N_IMG
idf = lambda g: np.log(I) - np.log(max(1, df[g]))
for omega in ("all", "order"):
    viol = up = down = 0
    for c, rs in zip(CANDS, REFS):
        hc = ngrams(c)
        for r in rs:
            hs = ngrams(r)
            for o in range(1, 5):
                a = {k: v for k, v in hc.items() if len(k) == o}; b = {k: v for k, v in hs.items() if len(k) == o}
                na = np.sqrt(sum((v * idf(k)) ** 2 for k, v in a.items()))
                nb = np.sqrt(sum((v * idf(k)) ** 2 for k, v in b.items()))
                if na == 0 or nb == 0:
                    continue
                Lc = sum(hc.values()) if omega == "all" else sum(a.values())
                Ls = sum(hs.values()) if omega == "all" else sum(b.values())
                code_t = sum(min(v, b.get(k, 0)) * b.get(k, 0) * idf(k) ** 2 for k, v in a.items()) / (na * nb)
                pap_t = sum(min(v / Lc, b.get(k, 0) / Ls) * b.get(k, 0) / Ls * idf(k) ** 2
                            for k, v in a.items()) / (na / Lc * nb / Ls)
                if (Lc >= Ls and pap_t < code_t - 1e-12) or (Lc <= Ls and pap_t > code_t + 1e-12):
                    viol += 1
                up += pap_t > code_t + 1e-12; down += pap_t < code_t - 1e-12
    pd = paper_cider_d(CANDS, REFS, omega)
    print(f"  normalizer over {omega:5s}: violations of the derived ordering {viol}; paper above code {up}, "
          f"below {down}; mean paper CIDEr-D {pd.mean():.4f} against code {code_d.mean():.4f}")

# ---------------------------------------------------------------- C3: one-image evaluation sets
print("\nC3  each image scored alone")
alone_c = [code_cider([CANDS[i]], [REFS[i]], "CIDEr")[0] for i in range(N_IMG)]
alone_d = [code_cider([CANDS[i]], [REFS[i]], "CIDEr-D")[0] for i in range(N_IMG)]
ext_d = np.array([code_cider([CANDS[i]], [REFS[i]], "CIDEr-D", "external", (POOL_DF, N_POOL))[0]
                  for i in range(N_IMG)])
print(f"  corpus IDF: nonzero CIDEr {sum(v != 0.0 for v in alone_c)}, nonzero CIDEr-D {sum(v != 0.0 for v in alone_d)}")
print(f"  external IDF ({N_POOL}-scene pool): zero CIDEr-D {int(np.sum(ext_d == 0.0))}, mean {ext_d.mean():.4f} "
      f"against {code_d.mean():.4f} in the 100-image set")

# ---------------------------------------------------------------- H4: evaluation-set composition
print("\nH4  the same caption in two 25-image evaluation sets")
cols = [s["colour"] for s in scenes[:N_IMG]]
diff = []
for i in range(N_IMG):
    same = [j for j in range(N_IMG) if cols[j] == cols[i]]
    others = [j for j in range(N_IMG) if cols[j] != cols[i]]
    pick = np.random.default_rng(1000 + i).permutation(others)
    mixed = [i] + [int(j) for c in COLOURS if c != cols[i] for j in [x for x in pick if cols[x] == c][:8]]
    s_same = code_cider([CANDS[j] for j in same], [REFS[j] for j in same])[1][same.index(i)]
    s_mix = code_cider([CANDS[j] for j in mixed], [REFS[j] for j in mixed])[1][0]
    diff.append(s_same - s_mix)
diff = np.array(diff)
print(f"  CIDEr-D same-colour set minus mixed set: min {diff.min():.4f}, max {diff.max():.4f}; "
      f"|difference| <= 1e-9 on {int(np.sum(np.abs(diff) <= 1e-9))} images; same higher on {int(np.sum(diff > 1e-9))}")

# ---------------------------------------------------------------- H5: repetition
print("\nH5  each caption repeated k times (references and IDF unchanged)")
u1 = code_cider(CANDS, REFS, "CIDEr")[2][:, 0]
_, cu1 = count_clip(CANDS, REFS, gaussian=False)
for k in (1, 2, 3, 4):
    rep = [" ".join([c] * k) for c in CANDS]
    _, sc, pn = code_cider(rep, REFS, "CIDEr")
    _, sd, _ = code_cider(rep, REFS, "CIDEr-D")
    _, cuk = count_clip(rep, REFS, gaussian=False)
    pos = cu1 > 0
    print(f"  x{k}: max |unigram CIDEr term - x1| {np.max(np.abs(pn[:, 0] - u1)):.1e}; "
          f"max |clipped unigram ratio - 1/k| {np.max(np.abs(cuk[pos] / cu1[pos] - 1 / k)):.1e}; "
          f"CIDEr mean {sc.mean():.4f} (above x1 on {int(np.sum(sc > code_c + 1e-12))}); "
          f"CIDEr-D mean {sd.mean():.4f} (below x1 on {int(np.sum(sd < code_d - 1e-12))})")

# ---------------------------------------------------------------- number of references, stemming
print("\nR   number of references (nested; IDF recomputed over the k-reference sets)")
for m in (1, 2, 3, 5, 10):
    r = [rs[:m] for rs in REFS]
    print(f"  {m:2d}: CIDEr {code_cider(CANDS, r, 'CIDEr')[0]:.4f}  CIDEr-D {code_cider(CANDS, r)[0]:.4f}  "
          f"10 x paper stemmed {10 * paper_cider(CANDS, r, stemmed=True).mean():.4f}")
ps, pu = paper_cider(CANDS, REFS, stemmed=True), paper_cider(CANDS, REFS)
print(f"S   10 x paper CIDEr, 10 references: stemmed {10 * ps.mean():.4f}, unstemmed {10 * pu.mean():.4f}; "
      f"stemmed higher on {int(np.sum(ps > pu + 1e-12))} of {N_IMG} images")

data check
  images 100, pool 200, references per image 10
  images per colour [25, 25, 25, 25], per count [15, 31, 28, 26]
  captioner reads count, colour and shape correctly on 100/100 images
  fewest words in a caption or reference 3
  most times a candidate word occurs in one reference 1
  example: candidate 'four blue triangle'; references 'the image shows four blue triangles', 'there are four blue triangles'

C1  code against the paper (100 images, 10 references, corpus IDF)
  x1: max |code CIDEr - 10 x paper (floor)| 1.3e-15; captions with an n-gram in no reference   0; mean 10 x paper floor 3.4408, limit 3.4408
  x2: max |code CIDEr - 10 x paper (floor)| 1.8e-15; captions with an n-gram in no reference 100; mean 10 x paper floor 3.1643, limit 1.8710
  x3: max |code CIDEr - 10 x paper (floor)| 8.9e-16; captions with an n-gram in no reference 100; mean 10 x paper floor 3.0312, limit 1.8710
  x4: max |code CIDEr - 10 x paper (floor)| 8.9e-16; captions with an n-gram in no reference 100; mean 10 x paper floor 2.9685, limit 1.8710
  max |code CIDEr-D - count-clipped form| 8.9e-16

C2  paper Eq. 4 against code CIDEr-D, per (image, reference, n) term before the penalty
  normalizer over all  : violations of the derived ordering 0; paper above code 0, below 1705; mean paper CIDEr-D 1.5899 against code 3.1266
  normalizer over order: violations of the derived ordering 0; paper above code 0, below 1705; mean paper CIDEr-D 1.8437 against code 3.1266

C3  each image scored alone
  corpus IDF: nonzero CIDEr 0, nonzero CIDEr-D 0
  external IDF (200-scene pool): zero CIDEr-D 0, mean 3.1474 against 3.1266 in the 100-image set

H4  the same caption in two 25-image evaluation sets
  CIDEr-D same-colour set minus mixed set: min -1.8638, max -0.1162; |difference| <= 1e-9 on 0 images; same higher on 0

H5  each caption repeated k times (references and IDF unchanged)
  x1: max |unigram CIDEr term - x1| 0.0e+00; max |clipped unigram ratio - 1/k| 0.0e+00; CIDEr mean 3.4408 (above x1 on 0); CIDEr-D mean 3.1266 (below x1 on 0)
  x2: max |unigram CIDEr term - x1| 0.0e+00; max |clipped unigram ratio - 1/k| 0.0e+00; CIDEr mean 3.1643 (above x1 on 0); CIDEr-D mean 1.5239 (below x1 on 100)
  x3: max |unigram CIDEr term - x1| 3.6e-15; max |clipped unigram ratio - 1/k| 1.1e-16; CIDEr mean 3.0312 (above x1 on 0); CIDEr-D mean 0.8160 (below x1 on 100)
  x4: max |unigram CIDEr term - x1| 0.0e+00; max |clipped unigram ratio - 1/k| 0.0e+00; CIDEr mean 2.9685 (above x1 on 0); CIDEr-D mean 0.3971 (below x1 on 100)

R   number of references (nested; IDF recomputed over the k-reference sets)
   1: CIDEr 2.0978  CIDEr-D 1.9170  10 x paper stemmed 4.4321
   2: CIDEr 2.3313  CIDEr-D 2.1469  10 x paper stemmed 4.7401
   3: CIDEr 2.5294  CIDEr-D 2.3357  10 x paper stemmed 5.0299
   5: CIDEr 2.8208  CIDEr-D 2.5716  10 x paper stemmed 5.4267
  10: CIDEr 3.4408  CIDEr-D 3.1266  10 x paper stemmed 6.0133
S   10 x paper CIDEr, 10 references: stemmed 6.0133, unstemmed 3.4408; stemmed higher on 100 of 100 images

The data check confirms the set-up: the captioner reads count, colour and shape correctly on every image, so its only error is the planned agreement error.

C1 held. The code’s CIDEr and ten times the paper’s Eq. 3 agree to about $10^{-15}$, for the captions and for their repetitions, as long as an n-gram missing from every reference gets the code’s floor. The two readings of the undefined IDF coincide for the unrepeated captions, none of which has such an n-gram. For the repeated ones they part: every repeated caption contains n-grams across the join (“triangle four”), which no reference has, and under the limit reading these send the cosines of orders 2 to 4 to zero and leave only the unigram term, which repetition does not change (H5a). That is why the limit reading prints the same 1.8710 for every $k \ge 2$. So “code CIDEr is 10 times paper CIDEr” is true only under the floor reading and without stemming. The independent count-clipped form matches the code’s CIDEr-D to $10^{-15}$, so clipping the weight vectors in the code is clipping counts. C1 held for that check as well.

C2 held. No term breaks the derived ordering under either reading of the normalizer. Every term where the two differ has the paper’s Eq. 4 below the code. In our reading, this follows from the construction: every candidate has three words and no reference has fewer, so the candidate’s normalizer is never the larger one. The paper’s equation and its text therefore can give different numbers whenever caption and reference lengths differ, and here the mean CIDEr-D is roughly half to three fifths of the code’s (1.5899 or 1.8437 against 3.1266, depending on the reading of the normalizer).

C3 held. Scored alone with corpus IDF, every image gets exactly 0 from both scorers, as the README warns. With document frequencies from the separate pool, no image scores 0, and the mean is close to the 100-image value.

H4 held. Every image’s CIDEr-D changes with the company it is scored in, by between 0.12 and 1.86. The direction is stated from the output, not predicted: on all 100 images the same-colour set gives the lower score. By construction, in that set the colour word occurs in the references of every image and so has IDF 0, but we do not derive the size or direction of the total change from it.

H5a, H5b and H5c held. Repeating a caption leaves its unclipped unigram term exactly unchanged, the cosine being blind to scale, and clipping alone divides that term by exactly $k$. The full CIDEr-D drops below the unrepeated score on every image for each $k$ (mean 3.1266 to 1.5239, 0.8160 and 0.3971), although at $k = 2$ the six-word caption is closer in length than the three-word one to every reference of five words or more, so for those references the Gaussian factor alone rises. Unclipped CIDEr also fell with repetition here and was above its unrepeated value on no image; the paper’s statement that basic CIDEr rewards repetition “in some cases” is not contradicted by this, since these short template captions are only one case.

R and S. On these data all three scores rise from one reference to ten. Each row is a separate evaluation with its own IDF, so the rise mixes more references with different weights; the paper’s finding about references concerns accuracy against people, which synthetic captions cannot measure. Stemming raises the paper’s CIDEr on every image, from 3.4408 to 6.0133 on the ×10 scale; in our reading, mainly because it turns “circles” in the references into the captioner’s “circle”, although it also changes the document frequencies. This is the error that CIDEr-D’s removal of stemming is meant to charge for.

Related Metrics

• BLEU is clipped n-gram precision with a brevity penalty, taking the maximum over references; BLEU gives its definition, parameters and implementations.
• METEOR aligns words by exact match, stem and synonym, computes a recall-weighted F-measure and keeps the best reference; it was CIDEr’s closest competitor in the paper’s experiments and was more accurate with five references on PASCAL-50S.
• ROUGE variants measure n-gram recall (ROUGE$_N$), longest common subsequence (ROUGE$_L$) or skip-bigrams (ROUGE$_S$); ROUGE$_L$ and ROUGE$_S$, as the CIDEr paper describes them, return the maximum over references.
• SPICE and CLIPScore, compared with CIDEr in Image Captioning, match scene-graph tuples and image–text embeddings instead of n-grams.