Show and Tell: A Neural Image Caption Generator

“Show and Tell: A Neural Image Caption Generator” by Oriol Vinyals, Alexander Toshev, Samy Bengio and Dumitru Erhan of Google was posted to arXiv in November 2014 and presented at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) in 2015. Its Neural Image Caption (NIC) model is one network trained end to end: a pretrained CNN encodes the image and an LSTM language model decodes the sentence, as in neural machine translation with an image in place of the source sentence. A journal version in IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI, 2017), “Show and Tell: Lessons Learned from the 2015 MSCOCO Image Captioning Challenge”, adds the changes that took the model to a shared first place in that challenge. The task and later methods are covered in image captioning.

Problem

The paper presents description as harder than classification, since a sentence must also say how objects relate and what they are doing. Earlier generation systems stitched components together: Farhadi et al. turned detections into a triplet of scene elements and filled templates, and Kulkarni et al. used a richer graph of detections, again with templates; the authors judge both hand-designed and rigid. Ranking systems embedded images and sentences in a common space and retrieved the closest stored description, which by the paper’s argument cannot describe a new composition of objects seen separately in training.

Kiros et al. had predicted the next word from the image and previous words with a feedforward network, and Mao et al. with a recurrent one, which the authors call very similar to their proposal; NIC differs in using a more powerful recurrent network and in giving the image directly to it. The inspiration was machine translation, where an encoder–decoder pair of RNNs maps a source sentence through a fixed-length vector to its translation (Sutskever et al., 2014); the paper swaps the encoder for a CNN.

Contribution

The paper lists three contributions:

• A single trainable system, one network trained with stochastic gradient descent.
• Pretrained components for vision and language, which can use larger corpora.
• Large gains. BLEU-1 rose from 25 to 59 on PASCAL, against about 69 for humans, from 56 to 66 on Flickr30k and from 19 to 28 on SBU.

The CVPR abstract adds BLEU-4 27.7 on COCO; the TPAMI version adds the lessons from the 2015 COCO challenge.

Method

Objective. Training maximizes the log probability of the correct description given the image, factorized word by word with the chain rule. The loss for one pair is the negative log likelihood of each correct word, $L(I, S) = -\sum_{t=1}^{N} \log p_t(S_t)$.

Decoder. The recurrent function is an LSTM whose gates are sigmoids of the input $x_t$ and previous output $m_{t-1}$, with output $m_t = o_t \odot c_t$ feeding a softmax over the vocabulary. Unrolled over a sentence $S = (S_0, \dots, S_N)$, with $S_0$ a start word and $S_N$ a stop word, the model is

$$x_{-1} = \text{CNN}(I), \qquad x_t = W_e S_t, \qquad p_{t+1} = \text{LSTM}(x_t), \quad t \in \{0, \dots, N-1\},$$

where $S_t$ is a one-hot word vector and $W_e$ a learned word embedding, so image and words share one input space.

The image enters once. The image is fed only at $t = -1$. Feeding it at every step gave worse results, which the authors attribute to the network exploiting image noise and overfitting. Show, Attend and Tell later noted this difference: the models of Kiros et al. and Mao et al. see the image at every step (Xu et al., 2015).

Image encoder. arXiv version 1 (November 2014) follows GoogLeNet, the ILSVRC 2014 classification winner (Szegedy et al., 2015). Version 2 (April 2015) and the CVPR text use a network with batch normalization (Ioffe and Szegedy, 2015), then the best on that task.

Training. The CNN was initialized from an ImageNet-pretrained model, which helped generalization a lot, and its weights were left unchanged because changing them hurt; the loss is minimized with respect to the LSTM, the word embeddings and the top layer of the image embedder, which we read as a trained projection on a frozen network. The TPAMI version says only the LSTM was trained, and the im2txt README describes a single trainable layer on a fixed encoder, consistent with that reading. Training used SGD with a fixed learning rate and no momentum, 512 dimensions for both embeddings and LSTM memory, and a vocabulary of words appearing at least five times. Dropout and ensembling gave a few BLEU points and are used in all reported results.

Inference. Captions came from beam search with a beam of 20; greedy decoding (beam 1) lowered results by 2 BLEU points on average.

Results

The experiments used PASCAL VOC 2008 (1,000 test images, no training set), Flickr8k, Flickr30k, COCO and SBU (one million Flickr images with their owners’ noisier captions). For COCO, 4,000 random validation images were held out as a test set, COCO-4k.

COCO development set	BLEU-4	METEOR	CIDEr
NIC	27.7	23.7	85.5
Random	4.6	9.0	5.1
Nearest neighbor	9.9	15.7	36.5
Human	21.7	25.2	85.4

For its BLEU-1 table the paper scores each human caption against the other four, corrected for having four references instead of five; it does not say whether the human row above was computed the same way. On the official COCO test set, scored through the evaluation server, NIC reached BLEU-4 27.2.

• BLEU-1 elsewhere. Besides the gains listed above (PASCAL with the COCO model), NIC scored 63 on Flickr8k against a previous 58 and a human 70.
• Transfer. Training on Flickr30k (about four times more data) scored 4 BLEU points higher than training on Flickr8k itself; training on COCO, “all the BLEU scores” fell by 10 points, which the authors put down to different collection and vocabulary. The CVPR text names no test set for these two runs; arXiv v1 reports the Flickr30k run on Flickr8k (BLEU-1 67 against 63) and gives BLEU-1 54 for the COCO run. Transferred from Flickr30k, PASCAL BLEU-1 was 53 instead of 59, and the COCO model scored 16 on SBU against 28 in domain.
• Novelty. The best beam candidate already appeared verbatim in the training set 80% of the time; about half of the top 15 candidates were novel.
• Ranking. Against MNLM, a ranking-trained Kiros et al. model, NIC had higher recall@1 on Flickr8k (20 against 18 for image annotation, 19 against 13 for image search); on Flickr30k it was lower for annotation (17 against 23) and level for search (17).
• Human ratings. Two Mechanical Turk workers rated each caption from 1 (unrelated) to 4 (no errors). NIC averaged 2.37 on Flickr8k, 2.45 on PASCAL and 2.72 on 1,000 COCO images, against 2.08 for an earlier system and 3.89 for the reference captions on Flickr8k. The authors conclude that BLEU, on which NIC beat human captions on COCO, misses this gap.

arXiv revisions. Version 1 reported BLEU-1 only (67 on COCO) plus BLEU-1 to BLEU-3 on Flickr8k, with a previous Flickr30k best of 55 and SBU 27. Version 2, which the CVPR text matches, switched to BLEU-4 and added METEOR and CIDEr as metrics that correlate better with human judgment.

The 2015 COCO Challenge

In the COCO captioning challenge of spring 2015, as TPAMI describes it, teams submitted at most five times to an evaluation server and human judges chose the winners. The organizers ranked teams by CIDEr, so the authors also selected models with it. The journal lists five changes and their BLEU-4 gains:

• A better image model (+2). The CVPR submission used GoogLeNet; the batch-normalized network lowered ImageNet top-5 error from 6.67% to 4.8%. The journal says the original experiments switched to it, and the CVPR text already describes it, so by our reading this step predates the published conference paper.
• Fine-tuning the CNN (+1). About 500K steps with the CNN frozen, then 100K jointly, took over three weeks on one Nvidia K20; joint training from the start corrupted the CNN. Afterwards the authors found many captions with correct colors, information they suggest ImageNet-trained features had discarded.
• Scheduled sampling (up to +1.5). A curriculum that gradually replaces the true previous word with the model’s own prediction during training (Bengio et al., 2015) targets the mismatch between training and inference.
• Ensembling (+1.5). The submission combined 5 models trained with scheduled sampling and 10 with fine-tuning.
• A smaller beam (more than +2). Selected by CIDEr, the best beam size was 3, not 20, and copied training captions fell from 80% to 60%; the authors read this as overfitting or as likelihood misaligned with human judgment. No other change raised CIDEr as much.

The refined model, NICv2, scored BLEU-4 32.1, METEOR 25.7 and CIDEr 99.8 on the development set. The text says BLEU-4 rose by 8 points since the CVPR submission, the sum of Table 8, while the development-set score rose by 4.4 points and the test-server score from 27.2 to 30.9; the paper does not reconcile these. One possible reading: the 8 points start from the GoogLeNet submission, for which neither arXiv version reports COCO BLEU-4. On the test server with five references per image, the Google entry ranked first by CIDEr (0.943), ahead of MSR Captivator (0.931). In the human evaluation, which decided the winners, it tied for first with Microsoft Research; its captions were judged as good as or better than a human caption 27.3% of the time and passed the Turing test 31.7% of the time; human captions scored 63.8% and 67.5%. With 40 references per image, human captions ranked 13th of 16 entries by BLEU-4 and 3rd by METEOR. The journal also released a TensorFlow implementation, im2txt, whose README uses Inception v3, a later encoder than either paper.

Impact

Later captioning papers compared against Show and Tell. Show, Attend and Tell (Xu et al., 2015) listed it in its results tables and argued that the single top-layer vector most such models used can lose information that attention over lower-level features keeps. Self-critical sequence training (Rennie et al., 2017) compared its four-model ensemble with NIC’s 15 models with fine-tuned CNNs. Conceptual Captions (Sharma et al., 2018) based its RNN captioner on Show and Tell, alongside transformer models that scored higher.

Limitations

The papers state several limits themselves:

• Copying. Most top captions reproduced training sentences, which the authors put down to small datasets.
• Metric mismatch. NIC beat humans on BLEU-4 and tied on CIDEr but was rated far below them, and the beam-size result suggests higher likelihood did not mean better captions.
• Frozen features. Until fine-tuning, ImageNet-trained features dropped attributes such as color, by the journal’s account.

Later work added two: one vector must hold everything a caption might mention (Xu et al., 2015), and maximum-likelihood training never exposes the decoder to its own mistakes (Rennie et al., 2017).

What Came After

Attention, CIDEr optimization with reinforcement learning, transformer decoders and vision–language pretraining replaced each part of NIC in turn, as image captioning traces. In our reading, the core formulation has stayed: an image encoder conditions an autoregressive language model trained to maximize the likelihood of the reference caption. Current vision–language models keep it, with a far larger pretrained decoder and image tokens in place of a single vector.