Image–Text Retrieval

Image–text retrieval (also cross-modal retrieval or image–text matching) ranks a collection in one modality by its relevance to a query in the other. In text-to-image retrieval a sentence such as “a dog catching a frisbee on a beach” ranks a gallery of photographs; in image-to-text retrieval an image ranks a pool of captions. It powers search by description and is a standard benchmark of multimodal learning, because a ranking can be scored automatically. This article builds on CLIP, the best-known dual encoder.

Problem Definition

Given a gallery $\mathcal{G} = \{x_1, \dots, x_N\}$ of images and a text query $t$, the system computes a score $s(x, t)$ for every candidate and returns the gallery sorted by decreasing score; image-to-text retrieval swaps the roles. The task is to learn $s$ so that relevant pairs outrank irrelevant ones. Hodosh, Young and Hockenmaier (2013) framed image description this way, as ranking a pool of captions by analogy with image search, and argued that ranking-based evaluation could be fully automated.

Input and Output

The input is a query and a gallery of the other modality: thousands of items in benchmarks, up to billions in deployed search. The output is a ranked list, usually truncated to the top $K$. Benchmarks treat a caption as relevant only to the image it was written for.

Variants

• Direction. Text-to-image (written IR or t2i) and image-to-text (TR or i2t); papers report both.
• Zero-shot versus fine-tuned. A pretrained model is evaluated directly or after training on the benchmark’s training split. The labels are used loosely: ALBEF and BLIP report “zero-shot” Flickr30K results with models fine-tuned on COCO (Li et al., 2021; Li et al., 2022).
• Other modalities. The same machinery serves text–video and text–audio retrieval, outside this article’s scope.

Challenges

• Many-to-many relevance. A caption fits many images, yet benchmarks mark one pairing as correct, so correct results count as errors.
• Fine-grained distinctions. Counts, attributes, spatial relations and word order separate near-duplicates; one pooled vector per image may not keep them.
• Cost at scale. Every query is compared with the whole gallery, which rules out expensive scoring unless candidates are pruned first.

Approaches

Joint embeddings with ranking losses

DeViSE (Frome et al., 2013) mapped CNN image features linearly into a word-embedding space with a hinge rank loss, an early visual–semantic embedding. Sentence-level models followed, with encoders $f$ and $g$, cosine score $s(i, c)$ between $f(i)$ and $g(c)$, and a bidirectional triplet ranking loss summed over the mini-batch negatives. VSE++ (Faghri et al., 2018) kept only the hardest negative in each direction:

$$\ell_{\text{MH}}(i, c) = \max_{c'} \big[\alpha + s(i, c') - s(i, c)\big]_+ + \max_{i'} \big[\alpha + s(i', c) - s(i, c)\big]_+,$$

with margin $\alpha$, $[x]_+ = \max(x, 0)$ and the maxima taken over the mini-batch. The motivation: the closest negative alone decides R@1. With a fine-tuned ResNet152 image encoder, VSE++ reached COCO 1K R@1 of 64.6 for caption retrieval and 52.0 for image retrieval. Softmax losses weight hard negatives too, but smoothly (contrastive learning).

Region–word alignment

Karpathy and Fei-Fei (2015) scored an image–sentence pair by aligning words with detected regions: each word takes its best-matching region and the scores are summed. SCAN (Lee et al., 2018) replaced the hard maximum with stacked cross attention: features of the 36 most confident regions from a Faster R-CNN trained on Visual Genome (appendix of the arXiv version, 1803.08024) attend over words (or words over regions), and the pooled relevances give the score. The score depends on both inputs, so no single cached vector per image suffices. SCAN reported Flickr30K R@1 of 67.4 (caption retrieval) and 48.6 (image retrieval) for its ensemble.

Dual encoders

A dual encoder (also two-tower model) embeds images and texts independently, $s(x, t) = \mathbf{a}(x)^\top \mathbf{b}(t)$ with $\ell_2$-normalized embeddings. The gallery is encoded once offline; a query costs one encoder pass and $N$ dot products, or fewer with an index. CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) trained such encoders contrastively on 400 million and 1.8 billion web pairs and evaluated them on retrieval zero-shot. For CLIP, prepending “a photo of” to each caption raised zero-shot R@1 by 1 to 2 points (arXiv appendix E.1). Fine-tuned, ALIGN stayed a dual encoder with the same loss.

Cross-encoders and retrieve-then-rerank

A cross-encoder feeds the image and the text together through a fusion network, typically a transformer with cross-attention, and outputs a match probability, the image–text matching (ITM) score, also a vision–language pretraining objective. Token-level interaction makes it more accurate, but ranking a gallery of $N$ images takes $N$ joint forward passes per query. ALBEF’s authors note that UNITER, OSCAR and VILLA computed the ITM score for every pair (Li et al., 2021).

ALBEF combined both: the dual-encoder image–text contrastive (ITC) similarity selects the top $k$ candidates, and the ITM head reranks only those. On fine-tuned Flickr30K, reranking with $k = 16$ raised mean recall (the average of R@1, R@5 and R@10) from 97.30 to 98.60 for caption retrieval and from 90.95 to 93.64 for image retrieval; $k = 128$ or $256$ changed little. BLIP adopted the procedure with $k = 256$ for COCO and $k = 128$ for Flickr30K (Li et al., 2022).

Late interaction

Late interaction sits between the two. FILIP (Yao et al., 2022), adapting the ColBERT text-search model, encodes each side separately into token embeddings and averages, over text tokens, each token’s maximum similarity to the image patches (and vice versa). Representations can still be precomputed offline, at the price of storing a vector per token.

Reported results

Recall@1 as printed by each paper; Flickr30K uses its 1K test set, COCO the 5K test set. “Fine-tuned” means trained on the benchmark’s training split.

Model	Setting	Flickr30K i2t	Flickr30K t2i	COCO 5K i2t	COCO 5K t2i	Source
Karpathy & Fei-Fei (BRNN)	trained on each dataset	22.2	15.2	16.5	10.7	Karpathy & Fei-Fei (2015), Table 1
VSE++ (ResNet, FT)	fine-tuned	52.9	39.6	41.3	30.3	Faghri et al. (2018), Tables 1, 3
SCAN (ensemble)	fine-tuned	67.4	48.6	50.4	38.6	Lee et al. (2018), Tables 1, 2
CLIP	zero-shot	88.0	68.7	58.4	37.8	Radford et al. (2021), arXiv Table 13
ALIGN	zero-shot	88.6	75.7	58.6	45.6	Jia et al. (2021), Table 1
ALIGN	fine-tuned	95.3	84.9	77.0	59.9	Jia et al. (2021), Table 1
ALBEF (14M images)	fine-tuned, rerank	95.9	85.6	77.6	60.7	Li et al. (2021), Table 2
BLIP (ViT-L, 129M images)	fine-tuned, rerank	97.4	87.6	82.4	65.1	Li et al. (2022), Table 5

Training data differ by orders of magnitude between rows, so this is a trajectory, not a controlled comparison.

Datasets and Benchmarks

• Flickr30K (Young et al., 2014): 31,783 Flickr photographs of everyday activities, events and scenes with 158,915 crowdsourced captions, five per image, extending the 8,092-image Flickr8K of Hodosh et al. (2013).
• MS-COCO captions: about five per image (COCO). Karpathy and Fei-Fei (2015) held out 5,000 images each for validation and testing on COCO and 1,000 each on Flickr30K, and reported COCO on the 5K test set and a 1,000-image subset, noting that Recall@K depends on test-set size. Later papers report COCO 1K as the average over five folds of 1,000 test images (Faghri et al., 2018) beside COCO 5K.
• ECCV Caption (Chun et al., 2022): five image–text matching models proposed and humans verified missing positives for a subset of COCO test queries, giving 1,261 image queries with 17.9 positive captions on average (originally 5) and 1,332 caption queries with 8.5 positive images (originally 1). Chun et al. describe the earlier Crisscrossed Captions (CxC) as focused on caption-to-caption similarity.

Evaluation Metrics

With one known correct item per query (or several, all counted), let $r_q$ be the rank of the first correct item for query $q$ among $Q$ queries:

$$\text{R@}K = \frac{100}{Q} \sum_{q=1}^{Q} \mathbb{1}[r_q \le K], \qquad \text{Med } r = \operatorname{median}_q \, r_q .$$

R@1, R@5 and R@10 are reported for both directions, often with RSUM, the sum of the six recalls (maximum 600). Hodosh et al. (2013) used R@K and the median rank as automatic metrics for this task. Unlike recall in classification, R@K is the fraction of queries with a hit in the top $K$, not of relevant items found. For image-to-text retrieval with five correct captions, Karpathy and Fei-Fei (2015) used the rank of the closest one.

On the queries they annotated, Chun et al. (2022) found 3.6 times as many correct image-to-caption and 8.5 times as many caption-to-image associations as COCO’s labels record, and proposed mAP@R, a form of average precision over the top $R$ results where $R$ is the number of positives. In their human study mAP@R agreed with human judgments better than R@K, and model rankings by ECCV Caption mAP@R correlated only weakly with COCO 1K R@1 (Kendall’s $\tau = 0.47$).

Scale Considerations

• Embedding size and normalization. CLIP’s joint embedding has 512 to 1,024 dimensions depending on the model (arXiv Tables 19–20). For unit vectors, cosine, dot product and Euclidean distance rank identically, so any inner-product or $\ell_2$ index works. At 4 bytes per value, a billion 512-dimensional vectors take about 2 TB before compression (our arithmetic).
• Approximate nearest-neighbor search. Exact search compares the query with every vector. An inverted file clusters the gallery with k-means and scans only the lists of the clusters nearest the query; product quantization compresses each vector into a short code from per-subspace codebooks, and the IVFADC index combines the two (Jégou et al., 2011). The FAISS library implements these on GPUs (Johnson et al., 2021). Feature matching covers the same trade-off for local descriptors; the bag of visual words built from SIFT descriptors was the classical image-retrieval use of an inverted index.
• Latency. Each cross-encoder pass is a full transformer evaluation, so only a short candidate list is reranked (128 or 256 in BLIP).
• Hubness. In high-dimensional spaces some points appear among the nearest neighbors of disproportionately many others (Radovanović et al., 2010). Bogolin et al. (2022) found such hubs in cross-modal embeddings and improved retrieval without retraining by renormalizing similarities against a bank of example queries (QB-Norm).

Example: Retrieval Metrics, Reranking and an Inverted File

The script builds a gallery of 1,000 synthetic $16 \times 16$ RGB images, each a disk, square, ring or bar in red, green or blue, small or large, on the left or right, with five invented captions per image (“a large blue square on the left”). Stand-in embeddings replace trained encoders: attribute word vectors, plus per-image “detail” shared by an image and its captions, plus noise; they keep shape and color but nearly lose the side. The script computes R@K, median rank and RSUM in both directions, reranks each caption’s top 10 images with a toy cross-encoder that adds a term computed from the caption and the pixels together (color, size and side read from the foreground mask), and searches an inverted file of 32 k-means lists. It runs in under 2 seconds with NumPy 2.3.

import numpy as np

rng = np.random.default_rng(9)
S, N, CAPS, D = 16, 1000, 5, 64
SHAPES, COLORS = ["disk", "square", "ring", "bar"], ["red", "green", "blue"]
SIZES, SIDES = ["small", "large"], ["left", "right"]
yy, xx = np.mgrid[0:S, 0:S] + 0.5

def render(shape, color, size, side):
    """A 16x16 RGB image: one colored shape on a dark, noisy background."""
    r = (2.5 if size == "small" else 4.5) + rng.uniform(-0.3, 0.3)
    cx = (5.0 if side == "left" else 11.0) + rng.uniform(-1, 1)
    x, y = xx - cx, yy - 8 - rng.uniform(-1, 1)
    m = {"disk": x**2 + y**2 < r**2,
         "square": (abs(x) < 0.85 * r) & (abs(y) < 0.85 * r),
         "ring": (x**2 + y**2 < r**2) & (x**2 + y**2 > (r - 1.5) ** 2),
         "bar": (abs(x) < 0.5 * r) & (abs(y) < r + 1)}[shape]
    img = 0.1 + 0.05 * rng.standard_normal((S, S, 3))
    img[m, COLORS.index(color)] += 0.8
    return img

# The gallery: N images with random attributes, each with 5 invented captions
attrs = [(rng.choice(SHAPES), rng.choice(COLORS), rng.choice(SIZES), rng.choice(SIDES))
         for _ in range(N)]
images = np.stack([render(*a) for a in attrs])                    # [N, 16, 16, 3]
cap_img = np.repeat(np.arange(N), CAPS)                           # caption -> its image
print(f"image 0: {S}x{S} RGB; caption 0: 'a {attrs[0][2]} {attrs[0][1]} "
      f"{attrs[0][0]} on the {attrs[0][3]}'")

# Stand-ins for trained dual-encoder outputs. Both encoders see the same concepts
# (attribute word vectors) plus instance detail; the encoders mostly lose the side.
def normalize(v):
    return v / np.linalg.norm(v, axis=-1, keepdims=True)
vocab = {w: rng.standard_normal(D) / np.sqrt(D) for w in SHAPES + COLORS + SIZES + SIDES}
weight = {"shape": 1.0, "color": 1.0, "size": 0.7, "side": 0.15}
def concept(a):
    return sum(weight[k] * vocab[w] for k, w in zip(weight, a))
detail = 0.5 * rng.standard_normal((N, D)) / np.sqrt(D)           # what else the captions say
img_emb = normalize(np.stack([concept(a) for a in attrs]) + detail
                    + 0.6 * rng.standard_normal((N, D)) / np.sqrt(D))
cap_emb = normalize(np.stack([concept(attrs[i]) for i in cap_img]) + detail[cap_img]
                    + 1.2 * rng.standard_normal((N * CAPS, D)) / np.sqrt(D))

def ranks_t2i(sim):
    """Rank (1 = best) of each caption's own image among all N images."""
    own = sim[np.arange(len(sim)), cap_img]
    return 1 + (sim > own[:, None]).sum(1)

def ranks_i2t(sim):
    """Rank of the best-ranked of each image's 5 captions (Karpathy & Fei-Fei)."""
    simT = sim.T                                                  # [N images, 5N captions]
    best_own = simT.reshape(N, N, CAPS)[np.arange(N), np.arange(N)].max(1)
    return 1 + (simT > best_own[:, None]).sum(1)

def report(name, r):
    rk = [100 * (r <= k).mean() for k in (1, 5, 10)]
    print(f"{name:16s} R@1 {rk[0]:5.1f}  R@5 {rk[1]:5.1f}  R@10 {rk[2]:5.1f}  "
          f"median rank {np.median(r):.0f}")
    return sum(rk)

sim = cap_emb @ img_emb.T                                         # [5000, 1000] cosines
print(f"\n1. Dual encoder: {sim.size:,} dot products for {len(sim):,} caption queries")
rsum = report("image -> text", ranks_i2t(sim)) + report("text -> image", ranks_t2i(sim))
print(f"RSUM {rsum:.1f}")

# How many top-1 'errors' are images that the caption describes correctly?
top1 = sim.argmax(1)
wrong = top1 != cap_img
same = np.array([attrs[top1[q]] == attrs[cap_img[q]] for q in np.flatnonzero(wrong)])
print(f"text -> image top-1 misses: {wrong.sum()}, of which {same.sum()} retrieve an image "
      f"with every attribute the caption states")

# 2. A toy cross-encoder reranks the top 10: it reads the caption and the pixels together
def pixel_attrs(img):
    fg = img.max(2) > 0.5
    color = COLORS[int(img[fg].mean(0).argmax())]
    side = "left" if xx[fg].mean() < S / 2 else "right"
    size = "small" if fg.sum() < 27 else "large"
    return color, size, side

def cross_score(q, i):
    """Joint score of caption q and image i: one 'forward pass' per pair."""
    stated = attrs[cap_img[q]][1:]                                # color, size, side in caption
    agree = sum(s == p for s, p in zip(stated, pixel_attrs(images[i])))
    return sim[q, i] + 0.1 * (agree - 3)

K = 10
top = np.argsort(-sim, axis=1)[:, :K]
passes = 0
new_rank = ranks_t2i(sim)
for q in range(len(sim)):
    scores = [cross_score(q, i) for i in top[q]]
    passes += K
    order = top[q][np.argsort(scores)[::-1]]
    hit = np.flatnonzero(order == cap_img[q])
    if hit.size:
        new_rank[q] = hit[0] + 1                                  # beyond 10: unchanged
print(f"\n2. Rerank top-{K} with the cross-encoder: {passes:,} joint passes "
      f"(scoring every pair would take {sim.size:,})")
report("text -> image", new_rank)

# 3. Approximate search: an inverted file over k-means clusters of the gallery
C = 32
cent = img_emb[rng.choice(N, C, replace=False)]
for _ in range(25):                                               # Lloyd's k-means
    assign = (img_emb @ cent.T).argmax(1)                        # unit vectors: max cosine
    cent = normalize(np.stack([img_emb[assign == c].mean(0) if (assign == c).any()
                               else cent[c] for c in range(C)]))
lists = [np.flatnonzero(assign == c) for c in range(C)]
exact_top1 = sim.argmax(1)
print(f"\n3. Inverted file, {C} k-means lists of {min(map(len, lists))}-{max(map(len, lists))} images")
for nprobe in (1, 2, 4, 8, 32):
    probe = np.argsort(-(cap_emb @ cent.T), axis=1)[:, :nprobe]
    found_exact, found_own, cost = 0, 0, 0
    for q in range(len(cap_emb)):
        cand = np.concatenate([lists[c] for c in probe[q]])
        cost += C + len(cand)                                     # centroids + list members
        best = cand[(img_emb[cand] @ cap_emb[q]).argmax()]
        found_exact += best == exact_top1[q]
        found_own += best == cap_img[q]
    print(f"nprobe {nprobe:2d}: {cost / len(cap_emb):6.1f} distances per query "
          f"(brute force {N}), same top-1 as brute force {100 * found_exact / len(cap_emb):5.1f}%, "
          f"R@1 {100 * found_own / len(cap_emb):4.1f}")

image 0: 16x16 RGB; caption 0: 'a large blue square on the left'

1. Dual encoder: 5,000,000 dot products for 5,000 caption queries
image -> text    R@1  41.0  R@5  77.5  R@10  91.2  median rank 2
text -> image    R@1  42.8  R@5  76.0  R@10  88.3  median rank 2
RSUM 416.8
text -> image top-1 misses: 2859, of which 1610 retrieve an image with every attribute the caption states

2. Rerank top-10 with the cross-encoder: 50,000 joint passes (scoring every pair would take 5,000,000)
text -> image    R@1  51.8  R@5  85.0  R@10  88.3  median rank 1

3. Inverted file, 32 k-means lists of 4-90 images
nprobe  1:   75.3 distances per query (brute force 1000), same top-1 as brute force  92.5%, R@1 41.0
nprobe  2:  105.5 distances per query (brute force 1000), same top-1 as brute force  98.9%, R@1 42.6
nprobe  4:  166.7 distances per query (brute force 1000), same top-1 as brute force 100.0%, R@1 42.8
nprobe  8:  292.3 distances per query (brute force 1000), same top-1 as brute force 100.0%, R@1 42.8
nprobe 32: 1032.0 distances per query (brute force 1000), same top-1 as brute force 100.0%, R@1 42.8

Of the 2,859 top-1 misses, 1,610 return an image with all four attributes the caption names; were the captions just those words, these would be correct answers scored as errors, the false-negative problem in miniature. Reranking ten candidates per query lifted text-to-image R@1 from 42.8 to 51.8 with 50,000 joint evaluations, a hundredth of exhaustive scoring; R@10 cannot change, since only the top ten are reordered. Probing two of 32 lists found the brute-force top-1 for 98.9% of queries with about a tenth of the distance computations; probing all 32 costs more than brute force because of the centroid comparisons. Over seeds 0–9, dual-encoder text-to-image R@1 was 40.9–44.2, reranking added 8.1–10.1 points, and probing one list matched the brute-force top-1 for 88.6–93.8% of queries.

Applications

• Photo and stock-image search by description, and caption lookup for archives.
• E-commerce: finding products from a description or a photo, with the catalog embedded offline.
• Dataset curation. LAION-5B kept web image–text pairs only if a CLIP ViT-B/32 image–text cosine similarity passed a threshold (0.28 for English; details in CLIP), removing about 90% of candidates, and released nearest-neighbor indices for exploring the data (Schuhmann et al., 2022).

Open Problems

• Evaluation with incomplete labels. Single-positive R@K penalizes correct answers; corrected sets such as ECCV Caption cover only a subset of queries and were built with model proposals that can bias them toward the proposing models (Chun et al., 2022).
• Saturation. Fine-tuned Flickr30K R@10 is 98.8–100 for ALBEF and BLIP (Li et al., 2022, Table 5), so the benchmark no longer separates strong models at large $K$.
• Compositional queries. Distinguishing “a red cube on a blue sphere” from its swap requires structure that a pooled embedding may not keep; by our reasoning, the bag-of-words limitation illustrated in multimodal learning applies to any score that ignores word order.