Flickr30k

Flickr30k is a corpus of 31,783 photographs from Flickr, each described by five independently written English sentences collected on Amazon Mechanical Turk. Peter Young, Alice Lai, Micah Hodosh and Julia Hockenmaier of the University of Illinois released it in 2014 in Transactions of the ACL (Young et al., 2014), as an extension of their 8,092-image Flickr8k (Hodosh, Young & Hockenmaier, 2013). In 2015 Plummer et al. added Flickr30k Entities: links between the phrases of the captions that mention the same entity, and boxes for the image regions they mention. With COCO captions, Flickr30k became one of the two standard benchmarks of image–text retrieval; it is also used for image captioning, and Flickr30k Entities is the standard benchmark for phrase grounding (visual grounding).

Purpose

Hodosh et al. (2013) wanted captions of a particular kind. Captions that owners write for their own photographs rarely say what the image shows: Young et al. (2014) contrast a caption naming the trip on which a photo was taken with one saying that three people are setting up a tent. Annotators who knew nothing about the circumstances of a photo, asked to describe the people, objects, scenes and activities in it, would instead produce conceptual descriptions of what can be seen. Five such captions per image capture how differently one scene can be described (“man” or “bike rider”, “doing tricks” or “jumping”). The same paper framed image description as ranking: a system scores a pool of human-written captions for an image, or a pool of images for a caption, and the rank of the correct item can be scored automatically, unlike generated text.

Young et al. (2014) built the larger corpus for a linguistic goal. They defined the visual denotation of an expression as the set of images it describes, and built a denotation graph: a subsumption hierarchy in which captions are reduced by dropping optional phrases, extracting simpler constituents and replacing nouns by hypernyms (“a poodle runs on the beach” entails “a dog runs”), each node carrying the images whose captions generate it. Similarities computed from shared denotations served two semantic-inference tasks, an approximate entailment task built from the captions and the Semantic Textual Similarity task.

Flickr30k Entities (Plummer et al., 2015) added what neither corpus provided: which words refer to which image regions. Its authors argued that such region-to-phrase supervision was needed for models that compose descriptions instead of matching whole images to whole sentences.

Dataset Size

Release	Images	Text	Other annotations	Source
Flickr8k	8,092	5 captions per image	train/dev/test lists, human relevance judgments for the test set	Hodosh et al. (2013)
Flickr30k	31,783	158,915 captions, 5 per image	denotation graph	Young et al. (2014); official page
Karpathy’s JSON release	31,014	5 captions per image	29,000/1,014/1,000 split	Elliott et al. (2016); not checked against the files
Flickr30k Entities v1.0	31,783	the 158,915 captions	244,035 coreference chains, 275,775 boxes	Plummer et al. (2015, 2017); README

Flickr8k. Hodosh et al. (2013) give 8,092 images. The workshop paper that describes the collection (Rashtchian et al., 2010) reports 8,108 hand-selected images; neither paper explains the difference, and the experiments of 2013 split 8,000 of them.

Flickr30k. The paper and the official page agree on 31,783 images and 158,915 captions, exactly five per image. On the corpus’s 158,439 unique captions, the denotation graph has 1,749,097 nodes by the text and 1,749,096 by Table 1 of the same paper; 230,811 of them describe more than one image and 161 describe 1,000 or more (Table 1; “over 1,000” in the text) (Young et al., 2014).

Karpathy’s release. Karpathy and Fei-Fei (2015) distributed dataset JSON files for Flickr8k, Flickr30k and COCO on their project page, which later code loads for the split. Multi30K describes Flickr30K as 31,014 images with 145,000 training, 5,070 development and 5,000 test descriptions, which is 29,000, 1,014 and 1,000 images at five captions each (Elliott et al., 2016; the division is ours), and papers that use Karpathy’s split report 29,000 training images (BLIP: “29k/1k/1k”; Li et al., 2022). Multi30K does not name Karpathy’s files; that they hold these 31,014 images is our inference from the two sources. Why the release has 769 fewer images than the paper is not documented in any source read for this article.

Flickr30k Entities. The annotation identified 513,644 entity or scene mentions (3.2 per caption), linked into 244,035 coreference chains, and drew 275,775 boxes, 8.7 per image (Plummer et al., 2017). Both versions give 16.6 mentions and 7.8 chains per image where they describe the coreference task, and 7.7 chains per image in their statistics section, which is what 244,035 / 31,783 gives; 513,644 / 31,783 gives 16.2 mentions.

Classes

Flickr30k has no category labels. Its content is defined by how the images were chosen: the Flickr8k images came from six Flickr groups (strangers!, Wild-Child (Kids in Action), Dogs in Action (Read the Rules), Outdoor Activities, Action Photography and Flickr-Social (two or more people in the photo)), and were selected by hand to show people or animals, mainly dogs, performing actions rather than scenery (Rashtchian et al., 2010; Hodosh et al., 2013). The official page says the added images and captions focus on people involved in everyday activities and events. Hodosh et al. note that the images tend not to show well-known people or places.

Flickr30k Entities assigns each mention one or more of eight coarse types: people, clothing, body parts, animals, vehicles, instruments, scene and other, plus notvisual for mentions that refer to nothing in the image, which carry the null chain id 0 (README). Aggregated over the five captions, people are mentioned in 94.2% of the images, clothing in 69.9%, body parts in 28.0%, vehicles in 13.8%, animals in 12.0% and instruments in 4.3%; the scene in 79.7% and other objects in 91.8% (Plummer et al., 2017). Of the chains, 59,766 are people and 82,098 other; people chains average 1.95 boxes, because phrases such as “a family” name several individuals.

Annotation Types

Captions. Workers wrote one sentence per image, five images per task. Rashtchian et al. (2010) compared quality-control schemes on PASCAL images and found that a qualification test improved quality most, whereas refining annotations through follow-up tasks worked poorly; for the Flickr corpus they required the test and US residency and paid 10 US cents per five images. The 2013 paper adds that workers had to pass a short spelling and grammar test; its online appendix holds the instructions. Young et al. (2014) used the same guidelines and similar quality controls for the added images, correcting spelling mistakes and eliminating ungrammatical or non-descriptive sentences. Flickr8k captions average 11.8 words, and 11% of them contain no verb (Hodosh et al., 2013).

Denotation graph. Building the graph required spell checking, tokenization, part-of-speech tagging, chunking and rule-based corrections; the tagging fixes alone changed 27,784 captions (17%). Words are lemmatized in the released graph (official page).

Flickr30k Entities. Annotation ran in two stages, each split into small Mechanical Turk tasks (Plummer et al., 2015, 2017):

1. Coreference. Noun-phrase chunks found automatically in the five captions are the candidate mentions. Workers judged whether pairs of mentions refer to the same set of entities, with mentions of different coarse types assumed not coreferent, and a verification task checked every chain of more than one mention.
2. Boxes. For each chain, workers decided whether a box can be drawn or the mention refers to the scene or to nothing visible, drew a box for each instance (or one around an indistinguishable group such as a crowd), judged each box’s quality, and confirmed that every instance was covered.

Only trusted workers took part: at least 500 earlier tasks with 95% approval, a qualification test for each of the six tasks and sufficient accuracy on their first 30 items. Redundant boxes from overlapping chains were merged above an IoU of 0.8 (0.9 for “other”), and about 4,000 entities with conflicting judgments were reviewed again, the worst by the authors. The released files hold, per image, the captions with bracketed phrases ([/EN#<chain id>/<types> words]) and a PASCAL VOC-style XML file with boxes or scene and no-box flags per chain. As of the latest commit (December 2018), the Python parser returns boxes with 0-based coordinates.

Splits

Almost all later work tests on 1,000 images, but the training and validation sizes differ by paper, and whether the different 1,000-image test lists contain the same images cannot be told from the papers.

Split	Train	Validation	Test	Source
Flickr8k, official	6,000	1,000	1,000	Hodosh et al. (2013)
Flickr8k, Karpathy	the rest	1,000	1,000	Karpathy & Fei-Fei (2015)
Flickr30k, Karpathy paper	the rest	1,000	1,000	Karpathy & Fei-Fei (2015)
Flickr30k, Karpathy JSON	29,000	1,014	1,000	Elliott et al. (2016); BLIP (“29k/1k/1k”)
Flickr30k, Show and Tell	28,000	1,000	1,000	Vinyals et al. (2015)
Flickr30k Entities	29,783	1,000	1,000	Plummer et al. (2015, 2017)

Karpathy and Fei-Fei (2015) wrote that they used 1,000 images for validation, 1,000 for testing and the rest for training, consistent with Hodosh et al. and with Karpathy et al.’s earlier fragment-embedding paper. Plummer et al. state 29,783 training images, the remainder of the paper’s 31,783, and say their split is the one of Gong et al. (2014); the README provides the lists and notes that Flickr30k has been evaluated under multiple splits. VSE++ gives Flickr30k a standard training set of 30,000 images and follows Karpathy’s split (Faghri et al., 2018). Later papers that load Karpathy’s JSON report its counts, as BLIP does. Every split is image-disjoint by construction, since an image’s five captions travel with it.

Tasks

• Image–text retrieval in both directions, the main use; the protocol is given below.
• Image captioning: generating a sentence scored against the five references. Hodosh et al. (2013) had argued for ranking partly because BLEU and ROUGE correlated poorly with human judgments of captions on their data.
• Phrase localization (phrase grounding): localizing each annotated phrase of a caption in its image, on Flickr30k Entities.
• Semantic inference: the approximate textual entailment task and the denotation similarities of Young et al. (2014).
• Multimodal machine translation on Multi30K, which adds German translations and independent German descriptions (Elliott et al., 2016).
• Zero-shot evaluation of vision–language pretraining models such as CLIP, retrieving on the 1K test set without training on Flickr30k.

Evaluation

Retrieval protocol

The official pages offer no evaluation server or retrieval code; results come from code run by each group on the public test split. The protocol that Karpathy and Fei-Fei (2015) reported and that most later papers follow is:

• Test set. The 1,000 test images and their 5,000 captions. Unlike COCO, where a 1K and a 5K test set are both in use, Flickr30k results use only the 1K set.
• Image-to-text (image annotation, TR, i2t). Each of the 1,000 images queries the 5,000 captions. The query’s rank is that of the highest-ranked of its five captions, the closest correct result in Karpathy and Fei-Fei’s description.
• Text-to-image (image search, IR, t2i). Each of the 5,000 captions queries the 1,000 images, and its rank is that of the single image it was written for.
• Scores. R@1, R@5 and R@10 in each direction, the percentage of queries whose rank is at most $K$ (Recall@K, where the metric itself is defined), with the median rank and often rsum, the sum of the six recalls (at most 600). Plummer et al. (2015) state the same protocol: given the 1,000 test images and 5,000 sentences, a query counts when at least one correct match is in the top $K$.

The VSE++ evaluation code, a third-party implementation (not from the dataset’s authors), computes it as follows (i2t and t2i in evaluation.py at commit abe382f, read line by line and run on the example below):

• The arrays hold one image row per caption, $5N$ rows for $N$ images, with the five captions of image $i$ at rows $5i$ to $5i + 4$. i2t takes row $5i$ as the query and scores it against all $5N$ captions with a dot product; t2i keeps every fifth image row and scores each caption against the $N$ images. Both thus rely on captions being stored in groups of exactly five; the Flickr loader builds its list image by image from Karpathy’s JSON (data.py at the same commit).
• Ranks come from numpy.argsort of the scores, reversed, and are 0-based. For i2t the rank is the minimum over the five captions’ positions.
• R@$K$ is 100 times the fraction of ranks below $K$; the median rank is $\lfloor \operatorname{median} \rfloor + 1$ of the 0-based ranks and the mean rank their mean plus one. evalrank prints rsum and the mean of the three recalls per direction. Its five-fold option, five consecutive blocks of 1,000 images, is for COCO only.
• With cosine similarity, the evaluation is exhaustive: no reranking, no candidate pruning.

Two conventions around the protocol matter when numbers are compared. Model selection: VSE++ chose the training snapshot with the highest sum of recalls on the validation set (Faghri et al., 2018), and its loader for precomputed features truncates the development set to its first 5,000 captions (data.py). Rerankers: models such as ALBEF and BLIP rescore only the top $k$ candidates of a dual encoder with a cross-encoder, $k = 128$ on Flickr30k for BLIP (Li et al., 2022), so their ranks beyond $k$ are the dual encoder’s.

The original Flickr8k protocol was different (Hodosh et al., 2013): the 1,000 test and 1,000 development images were each paired with one arbitrarily chosen caption, so both directions ranked 1,000 items against 1,000 with one correct answer, and significance was tested (McNemar’s test for R@K, Fisher’s randomization test for the median rank). The five-caption protocol makes the two directions unequal: an image query has five chances against a five-times-larger pool, a caption query one chance.

Example: Recall@K with five captions per image

The script renders 1,000 synthetic $16 \times 16$ RGB images, each one shape (disk, square, triangle or ring) in one of four colors, small or large, on the left or right, and writes five captions per image. Like crowd workers who differ in specificity, each caption names the shape and each other attribute with probability 0.6. Stand-ins replace trained encoders: an image is embedded from what a crude reader recovers from its pixels (fill ratio and hole for the shape, mean foreground color, area, centroid), a caption from the words it contains, both plus a per-image “detail” vector and noise. The script computes the 1K-test protocol, repeats it with one caption per image counted correct as in Hodosh et al. (2013), and breaks text-to-image R@1 down by how much a caption says. Given the path of VSE++’s evaluation.py as an argument, it also extracts the official i2t and t2i functions unchanged (they run under Python 3 when npts is passed as an integer) and compares their ranks with its own. It runs in under 3 seconds with NumPy 2.3.

import ast, sys
import numpy as np

S, N, CAPS, D = 16, 1000, 5, 64
SHAPES, COLORS = ["disk", "square", "triangle", "ring"], ["red", "green", "blue", "yellow"]
SIZES, SIDES = ["small", "large"], ["left", "right"]
RGB = {"red": (1, 0, 0), "green": (0, 1, 0), "blue": (0, 0, 1), "yellow": (1, 1, 0)}
yy, xx = np.mgrid[0:S, 0:S] + 0.5

def render(rng, shape, color, size, side):
    """A 16x16 RGB image: one colored shape on a dark, noisy background."""
    r = (2.5 if size == "small" else 4.5) + rng.uniform(-0.3, 0.3)
    x = xx - (4.5 if side == "left" else 11.5) - rng.uniform(-0.7, 0.7)
    y = yy - 8 - rng.uniform(-1, 1)
    m = {"disk": x**2 + y**2 < r**2,
         "square": (abs(x) < 0.85 * r) & (abs(y) < 0.85 * r),
         "triangle": (y < 0.8 * r) & (y > -r) & (abs(x) < 0.5 * (y + r)),
         "ring": (x**2 + y**2 < r**2) & (x**2 + y**2 > (0.55 * r) ** 2)}[shape]
    img = 0.1 + 0.05 * rng.standard_normal((S, S, 3))
    img[m] += 0.8 * np.array(RGB[color])
    return img

def read_pixels(img):
    """What a crude image encoder recovers from the pixels alone."""
    fg = img.max(2) > 0.5
    rows, cols = np.flatnonzero(fg.any(1)), np.flatnonzero(fg.any(0))
    fill = fg.sum() / ((rows[-1] - rows[0] + 1) * (cols[-1] - cols[0] + 1))
    cy, cx = int(yy[fg].mean()), int(xx[fg].mean())
    shape = ("ring" if not fg[cy, cx] else "square" if fill > 0.9
             else "disk" if fill > 0.65 else "triangle")
    mean = img[fg].mean(0)
    color = min(COLORS, key=lambda c: np.abs(mean - 0.1 - 0.8 * np.array(RGB[c])).sum())
    return shape, color, "small" if fg.sum() < 30 else "large", "left" if cx < S / 2 else "right"

def build(seed):
    rng = np.random.default_rng(seed)
    attrs = [(rng.choice(SHAPES), rng.choice(COLORS), rng.choice(SIZES), rng.choice(SIDES))
             for _ in range(N)]
    images = np.stack([render(rng, *a) for a in attrs])            # [N, 16, 16, 3]
    # Five captions per image; like crowd workers, each states the shape and a
    # random subset of the other attributes ("a red disk", "a small red disk on the left")
    said = rng.random((N, CAPS, 3)) < 0.6                           # color, size, side mentioned?
    vocab = {w: rng.standard_normal(D) / np.sqrt(D) for w in SHAPES + COLORS + SIZES + SIDES}
    unit = lambda v: v / np.linalg.norm(v, axis=-1, keepdims=True)
    detail = 0.6 * rng.standard_normal((N, D)) / np.sqrt(D)        # what else the scene shows
    img_emb = unit(np.stack([sum(vocab[w] for w in read_pixels(im)) for im in images])
                   + detail + 0.6 * rng.standard_normal((N, D)) / np.sqrt(D))
    cap_words = [[attrs[i][0]] + [attrs[i][k + 1] for k in range(3) if said[i, j, k]]
                 for i in range(N) for j in range(CAPS)]
    cap_emb = unit(np.stack([sum(vocab[w] for w in ws) for ws in cap_words])
                   + np.repeat(detail, CAPS, 0)
                   + 0.9 * rng.standard_normal((N * CAPS, D)) / np.sqrt(D))
    return attrs, images, cap_words, img_emb, cap_emb

def ranks_i2t(img_emb, cap_emb):
    """0-based rank of the best-ranked of each image's five captions among all 5N."""
    sim = img_emb @ cap_emb.T                                       # [N, 5N]
    own = sim.reshape(len(img_emb), -1, CAPS)[np.arange(len(img_emb)), np.arange(len(img_emb))]
    return (sim > own.max(1, keepdims=True)).sum(1)

def ranks_t2i(img_emb, cap_emb):
    """0-based rank of each caption's single correct image among all N images."""
    sim = cap_emb @ img_emb.T                                       # [5N, N]
    own = sim[np.arange(len(sim)), np.arange(len(sim)) // CAPS]
    return (sim > own[:, None]).sum(1)

def recalls(r):
    return [100 * (r < k).mean() for k in (1, 5, 10)] + [np.floor(np.median(r)) + 1]

if __name__ == "__main__":
    attrs, images, cap_words, img_emb, cap_emb = build(0)
    print(f"{N} images of {S}x{S} RGB, {N * CAPS} captions; image 0 captions:")
    for ws in cap_words[:CAPS]:
        side = [w for w in ws if w in SIDES]
        adj = [w for w in ws if w in SIZES] + [w for w in ws if w in COLORS]
        print("    a", " ".join(adj + ws[:1]) + (f" on the {side[0]}" if side else ""))
    acc = np.mean([read_pixels(im) == a for im, a in zip(images, attrs)])
    print(f"pixel reader recovers all four attributes for {100 * acc:.1f}% of images\n")

    ri, rt = ranks_i2t(img_emb, cap_emb), ranks_t2i(img_emb, cap_emb)
    a, b = recalls(ri), recalls(rt)
    print("1K test protocol      R@1   R@5  R@10  MedR  queries")
    print(f"image -> text      {a[0]:5.1f} {a[1]:5.1f} {a[2]:5.1f} {a[3]:5.0f}  {len(ri):6d}")
    print(f"text -> image      {b[0]:5.1f} {b[1]:5.1f} {b[2]:5.1f} {b[3]:5.0f}  {len(rt):6d}")
    print(f"rsum {sum(a[:3]) + sum(b[:3]):.1f}")

    # Same embeddings, one caption per image counted correct (Hodosh et al. 2013 kept
    # one arbitrarily chosen caption per test image): 1,000 queries against 1,000.
    pick = np.random.default_rng(1).integers(0, CAPS, N)
    one = cap_emb.reshape(N, CAPS, D)[np.arange(N), pick]
    sim1 = img_emb @ one.T
    r1_i2t = (sim1 > np.diag(sim1)[:, None]).sum(1)
    r1_t2i = (sim1.T > np.diag(sim1)[:, None]).sum(1)
    a1, b1 = recalls(r1_i2t), recalls(r1_t2i)
    print("\none caption/image   R@1   R@5  R@10  MedR")
    print(f"image -> text      {a1[0]:5.1f} {a1[1]:5.1f} {a1[2]:5.1f} {a1[3]:5.0f}")
    print(f"text -> image      {b1[0]:5.1f} {b1[1]:5.1f} {b1[2]:5.1f} {b1[3]:5.0f}")

    # Text -> image R@1 by how much the caption says
    n_said = np.array([len(ws) for ws in cap_words])
    print("\ntext -> image R@1 by attributes stated in the caption")
    for k in range(1, 5):
        print(f"  {k} of 4: {100 * (rt[n_said == k] < 1).mean():5.1f}  ({(n_said == k).sum()} captions)")

    # Optional cross-check against the official VSE++ functions
    if len(sys.argv) > 1:
        src = open(sys.argv[1]).read()
        ns = {"numpy": np}
        for node in ast.parse(src).body:
            if isinstance(node, ast.FunctionDef) and node.name in ("i2t", "t2i"):
                exec(compile(ast.Module([node], []), sys.argv[1], "exec"), ns)
        rep = np.repeat(img_emb, CAPS, 0)        # VSE++ stores one image row per caption
        (o1, o2), (oi, ot) = [ns[f](rep, cap_emb, npts=N, return_ranks=True) for f in ("i2t", "t2i")]
        same = np.array_equal(o2[0], ri) and np.array_equal(ot[0], rt)
        print(f"\nVSE++ i2t {o1[0]:.1f} {o1[1]:.1f} {o1[2]:.1f} medr {o1[3]:.0f}, "
              f"t2i {oi[0]:.1f} {oi[1]:.1f} {oi[2]:.1f} medr {oi[3]:.0f}; identical ranks: {same}")

    # Spread over seeds
    gap, r1i, r1t = [], [], []
    for seed in range(10):
        _, _, _, ie, ce = build(seed)
        x, y = recalls(ranks_i2t(ie, ce))[0], recalls(ranks_t2i(ie, ce))[0]
        r1i.append(x); r1t.append(y); gap.append(x - y)
    print(f"\nseeds 0-9: i2t R@1 {min(r1i):.1f}-{max(r1i):.1f}, t2i R@1 {min(r1t):.1f}-{max(r1t):.1f}, "
          f"gap {min(gap):.1f}-{max(gap):.1f} points")

Output of python recall_flickr.py vsepp/evaluation.py, with evaluation.py from commit abe382f:

1000 images of 16x16 RGB, 5000 captions; image 0 captions:
    a large blue ring
    a large blue ring on the left
    a large ring on the left
    a large blue ring on the left
    a large blue ring on the left
pixel reader recovers all four attributes for 89.7% of images

1K test protocol      R@1   R@5  R@10  MedR  queries
image -> text       55.0  69.1  75.8     1    1000
text -> image       46.4  65.9  73.7     2    5000
rsum 385.9

one caption/image   R@1   R@5  R@10  MedR
image -> text       21.0  43.2  55.8     7
text -> image       44.0  65.3  73.4     2

text -> image R@1 by attributes stated in the caption
  1 of 4:  13.7  (313 captions)
  2 of 4:  25.0  (1410 captions)
  3 of 4:  48.6  (2219 captions)
  4 of 4:  80.0  (1058 captions)

VSE++ i2t 55.0 69.1 75.8 medr 1, t2i 46.4 65.9 73.7 medr 2; identical ranks: True

seeds 0-9: i2t R@1 52.2-56.6, t2i R@1 42.5-46.4, gap 7.1-11.4 points

The official functions return the same ranks as the vectorized version, so the 0-based ranks, the minimum over five captions and the median-rank formula above are what VSE++ computes. Both directions read the same similarity matrix, yet image-to-text R@1 is higher, by 7 to 11 points over the ten seeds. Counting one caption per image instead of five removes that advantage and reverses it, while text-to-image hardly changes: under the five-caption protocol an image needs only its best caption near the top, and the best is usually the most specific. The breakdown by caption content shows how much specificity matters. A caption naming only the shape fits about a quarter of the gallery and rarely finds its own image first; a caption naming all four attributes usually does. By our reasoning, generic captions of other images also act as hubs that outrank an image’s own caption when only one caption counts, which hurts image queries more than caption queries. In every Flickr30k result quoted in this article image-to-text R@1 exceeds text-to-image R@1; by our reasoning, the protocol, not only the models, contributes to that gap when an image’s captions differ in specificity.

Caption metrics

Karpathy and Fei-Fei (2015) computed BLEU, METEOR and CIDEr on Flickr8k, Flickr30k and COCO with the coco-caption code, each candidate scored against the image’s five reference captions. As that code computes them (pycocoevalcap at commit 3a9afb2, read line by line; it requires Python 2.7 and Java), candidates and references are first tokenized with the Stanford CoreNLP PTB tokenizer, then:

• BLEU-1 to BLEU-4 are computed over the whole test set, with the brevity penalty from the reference length closest to each candidate’s;
• METEOR runs the METEOR 1.5 jar, with changes noted in the README to aggregate statistics over the corpus;
• ROUGE-L is computed beside them;
• CIDEr is the CIDEr-D variant: counts clipped, a Gaussian length penalty with $\sigma = 6$, a factor of 10, and document frequencies counted over the reference sets of the images being evaluated, with the IDF denominator their number. A Flickr30k CIDEr therefore depends on which 1,000 images form the test set, not only on the captions;
• SPICE was added to the toolkit in April 2018.

The image captioning article defines the metrics and discusses how well they agree with human judgments. Karpathy and Fei-Fei (2015) reported Flickr30k BLEU-4 15.7, METEOR 15.3 and CIDEr 24.7 for their multimodal RNN on 1,000 test images, the last two in a note under Table 2. Most later captioning papers report COCO’s Karpathy test split instead (image captioning).

Phrase localization on Flickr30k Entities

Plummer et al. defined phrase localization as retrieval inside one image: each annotated phrase of a test caption queries a set of region proposals of its image, and the query counts at $K$ if a proposal among the top $K$ has intersection over union of at least 0.5 with the phrase’s ground-truth box. The two versions differ:

	Conference version (2015)	Journal version (2017)
Proposals	top 100 EdgeBox proposals	200 EdgeBox proposals, NMS at IoU 0.5
Phrases	parsed NP chunks exactly matching a ground-truth phrase; no match counts as a miss	ground-truth NP chunks, non-visual mentions ignored
Scores	R@1, R@10, R@100, and AP after NMS	R@1, R@5, R@10 only

The journal version dropped average precision because the boxes are sparse: many valid regions, especially body parts and clothing, lack boxes because no caption mentions them, a reporting bias that makes AP unreliable. Extracting the phrases automatically instead of using ground-truth chunks lowered recall by about 3%. Recall at the number of proposals (R@100 or R@200) is the upper bound set by the proposals.

A phrase with several boxes (“two men”) needs a rule. Plummer et al. (2015) merged a phrase’s boxes into one for training their CCA model. MDETR (Kamath et al., 2021, arXiv appendix) named the two conventions in use: any-box, under which a prediction is correct if it overlaps any of the phrase’s boxes enough, and merged-boxes, under which the ground truth is the smallest box enclosing them all. MDETR reported both, and had to fine-tune on merged boxes for the second; its appendix writes the criterion as IoU above 0.5, whereas Plummer et al. write at least 0.5. The dataset repository at commit 68b3d6f provides the split lists and MATLAB and Python parsers but no evaluation script. The project page points instead to the authors’ later pl-clc code for evaluating phrase localization (MATLAB, commit 41ee493, read line by line; not run, as it needs the dataset): it drops phrases without a box, replaces a phrase’s boxes by their enclosing box, as under merged boxes, and counts the top-ranked proposal correct at IoU ≥ 0.5, with box sides measured in inclusive pixels (width $x_2 - x_1 + 1$); it reports R@1 and the proposals’ upper bound.

License

The terms below are as stated on the official pages, as of October 2026.

• Images. The Flickr30k page gives the images through a request form and states that they come from Flickr, that their use must abide by the Flickr Terms of Use, that the authors do not own their copyright, and that they are provided only for researchers and educators for non-commercial research or educational purposes. The Flickr8k page states the same for Flickr8k. Young et al. (2014) write that almost all of the images they added are under a Creative Commons licence; the individual licences are not listed on the pages.
• Flickr8k captions. The distributable version’s page says the captions are released under a “Creative Commons Attribution-ShareAlike license”, but the link points to the Attribution 3.0 licence (CC BY 3.0) without ShareAlike. The two disagree; the page does not resolve it.
• Flickr30k captions and the denotation graph. The page names no licence for them; they come with the images through the same request form. The terms for the text are therefore unclear.
• Flickr30k Entities. The repository has no licence file. Its README repeats the Flickr terms and the non-commercial research and educational purpose, and asks users to cite both papers; no licence is named for the annotations themselves.
• Karpathy’s JSON files. The archived project page offers them for download without stating terms.
• Multi30K. The 2016 paper releases its German data under CC BY-NC-SA 4.0 (Elliott et al., 2016).

Limitations and Bias

People in the images. Flickr30k is a dataset of people: they are mentioned in 94.2% of its images (Plummer et al., 2017), and Flickr8k’s sources include groups such as strangers! and Wild-Child (Kids in Action). The dataset papers describe how images were chosen and that most added ones are Creative Commons licensed; none describes contact with the people photographed or a removal process (our reading of Rashtchian et al., 2010, Hodosh et al., 2013 and Young et al., 2014).

Stereotypes in the captions. Hodosh et al. (2013) wrote that their instructions yielded descriptions based on the image alone. Van Miltenburg (2016), in a workshop paper, examined a subset of Flickr30k by browsing images with their captions and argued that this assumption fails. He grouped unwarranted inferences, statements the image cannot support, into six kinds: activity, ethnicity, event, goal, relation and status or occupation. Examples include a woman and a man described as a worker scolded by her boss, dark-skinned people called African-American regardless of where the photo was taken, and older people with children called parents. He also measured linguistic bias in how race is marked for babies. Of the 697 images with “baby” in a caption, he classified 504 as showing white, 66 Asian and 36 black babies (73 showed no baby, 18 fit no category). White babies were marked in fewer than 1% of the images, Asian and black babies much more often: in the 14 images his text search found with a race-marked baby (two of them false positives), 60% of captions marked race for Asian babies, 40% for black and 20% for white (Table 1). The study is qualitative apart from this case and covers no other age group; gender stereotypes are illustrated by examples, not counted. He argued that models trained on the data may propagate its stereotypes.

Subjective language. Van Miltenburg, Morante and Elliott (2016) found negations (“no”, “not”, “nobody”) in 0.56% of Flickr30k descriptions; in 86.25% of the images with a negation only one of the five captions had one, which they read as a subjective choice of the annotator.

Caption and annotation errors. Despite the quality control, the Entities README lists, in an UNRELATED_CAPTIONS file, captions that do not relate to their images, and notes that the list is likely incomplete. Plummer et al. (2017) found chunking errors in fewer than 1% of mentions and coreference errors in fewer than 1% of chains, which with over 16 mentions and 7 chains per image means an error of some kind in about 8% of images. Their boxes are also incomplete by design: regions never mentioned have no box.

Missing positives. Retrieval scores treat a caption as correct only for its own image, though many captions fit other images. Hodosh et al. (2013) collected human relevance judgments for their Flickr8k test pool and found that their best system’s first caption was the original one for 8.3% of images, but described the image without errors for 11.0% by their expert judges; R@K is a lower bound. The false-negative studies of image–text retrieval concern COCO; none of the sources read for this article measures them on Flickr30k.

Scope and saturation. The captions are English, written by US workers (Rashtchian et al., 2010), and the 1K test set is small. The fine-tuned ALIGN, ALBEF and BLIP rows of BLIP’s Table 5 score R@10 of 98.6 to 100 in both directions (Li et al., 2022), so, by our reading, the benchmark now separates strong models mainly at R@1. Web-scale pretraining data can contain Flickr30k test images; see vision–language pretraining for the deduplication that some groups reported (benchmark overfitting, dataset bias).

Common Uses

• Fine-tuned and zero-shot image–text retrieval, reported beside COCO in vision–language papers such as CLIP and BLIP.
• Phrase grounding on Flickr30k Entities, and grounding data for pretraining: MDETR’s pretraining set combined Flickr30k with COCO and Visual Genome (Kamath et al., 2021).
• Captioning on Flickr8k and Flickr30k in the early neural models; the Show and Tell paper reported both.
• Multilingual captioning and multimodal translation through Multi30K.

Historical Milestones

• 2013: ranking as evaluation. On the Flickr8k one-caption protocol, Hodosh et al.’s best KCCA model (Tri5Sem) reached R@1 8.3 for annotation and 7.6 for search, median ranks 34 and 38 (Table 3).
• 2015: the Karpathy protocol. Karpathy and Fei-Fei’s BRNN alignment model scored Flickr30k R@1 22.2 (image annotation) and 15.2 (image search) on the five-caption 1K test, averaged over the five best validation models (Table 1).
• 2015–2017: phrase localization. Plummer et al.’s CCA baseline localized 25.30% of test phrases at R@1 with 100 proposals (2015, Table 2); with Fast R-CNN features, detector scores, a size prior and a color classifier, the journal version reached 50.89 with 200 proposals (2017, Table 5).
• 2018: hard negatives. VSE++ with a fine-tuned ResNet-152 reached Flickr30k R@1 52.9 for caption retrieval and 39.6 for image retrieval (Faghri et al., 2018, Table 3).
• 2021: zero-shot transfer. CLIP reached zero-shot R@1 88.0 for text retrieval and 68.7 for image retrieval, prepending “a photo of” to the captions, which raised R@1 by 1 to 2 points (Radford et al., 2021, arXiv Table 13). MDETR reached phrase grounding test R@1 84.3 under the any-box protocol (EfficientNet-B5 backbone) and 83.8 under merged boxes (ResNet-101), both pretrained on COCO, Visual Genome and Flickr30k (Kamath et al., 2021, Table 3).
• 2022: saturation. Fine-tuned BLIP with ViT-L reached R@1 97.4 and 87.6, with R@10 of 99.9 and 99.0 (Li et al., 2022, Table 5).

Related Datasets

• Flickr8k (above) and the 1,000 PASCAL VOC 2008 images with five captions each that the same group collected first (Rashtchian et al., 2010).
• COCO Captions: about 123,000 images with five captions each in Karpathy and Fei-Fei’s experiments, with the 1K and 5K test protocols; the main captioning benchmark.
• Multi30K: German translations of one caption per image and 155,070 independent German descriptions for the Flickr30k images (Elliott et al., 2016).
• Visual Genome: region descriptions, objects and relationships for 108,077 images (as tabulated by Plummer et al., 2017), a denser source of region–phrase pairs.
• SBU Captioned Photos: about a million Flickr images with the captions their owners wrote, larger but noisier, the kind of caption Hodosh et al. (2013) set out to avoid.
• ReferItGame, RefCOCO and RefCOCOg: referring expressions with one box each, the other grounding benchmarks (visual grounding).

Resources

• Flickr30k and Denotation Graph page: request form for the images and captions, the denotation graph and the entailment data.
• Flickr8k page: request form, paper and demos.
• Flickr30k Entities repository: annotations, split lists, parsers and the unrelated-captions list; the authors’ phrase-localization evaluation is in pl-clc.
• Karpathy’s project page (archived): the split JSON files.
• VSE++ evaluation code and coco-caption: third-party implementations of the retrieval and caption metrics, used by VSE++ and by Karpathy and Fei-Fei.
• There is no official leaderboard; current results appear in the papers that report them.