Neural Machine Translation by Jointly Learning to Align and Translate

“Neural Machine Translation by Jointly Learning to Align and Translate” is a paper by Dzmitry Bahdanau, then at Jacobs University Bremen, and Kyunghyun Cho and Yoshua Bengio of the Université de Montréal. It was posted to arXiv on 1 September 2014, revised six times, and presented as an oral at the International Conference on Learning Representations (ICLR 2015). The paper replaced the single fixed-length vector through which earlier neural translation models passed the whole source sentence: its decoder computes, for every word it writes, a fresh weighted average of per-word encoder states, with weights from a small network trained together with the translator. The authors called this network an alignment model and the operation a soft search; it is the additive form of the attention mechanism, the name the paper itself gives it in one paragraph. The paper is about machine translation and contains no vision experiments: every result is English-to-French translation. Its mechanism reached computer vision within months, in image captioning, and later through the transformer. This article covers the publication; the attention mechanism article explains attention as it is understood today, with its scoring functions and an example over image patches.

Problem

Neural machine translation was, in the paper’s words, a newly emerging approach. The paper names Kalchbrenner and Blunsom (2013), Sutskever, Vinyals and Le (2014) and Cho et al. (2014) as its proposers, and most of their models were encoder–decoders. In Cho et al.’s RNN Encoder–Decoder (EMNLP 2014), the encoder’s hidden state after the end-of-sentence symbol is a summary $\mathbf{c}$ of the whole input, and the decoder’s state and output distribution are conditioned on $\mathbf{c}$ at every step. That paper used the model to score phrase pairs inside a phrase-based system, and it also introduced a gated hidden unit with a reset gate and an update gate, which Bahdanau et al. reused. Sutskever et al. (NIPS 2014) used a four-layer LSTM to read the source into a vector and another to decode it.

Whether the vector was a bottleneck was an open question. Cho, van Merriënboer, Bahdanau and Bengio (SSST-8, 2014) analysed two encoder–decoders and found that translation quality degraded quickly as sentences grew longer and as the number of unknown words grew. They offered as the most obvious explanation that a fixed-length vector lacked the capacity for a long sentence, but their conclusion also suggested the decoder’s lack of representational power, since two very different encoders suffered alike. Sutskever et al. reported the opposite experience: their LSTM did not suffer on very long sentences, which they attributed to reversing the source word order. Bahdanau et al. took the first reading. Their introduction conjectures that the fixed-length vector is a bottleneck, especially for sentences longer than those seen in training, and cites Cho et al.’s analysis as evidence.

Contribution

• A decoder that searches the source. Instead of one vector $\mathbf{c}$, each target word $y_i$ gets its own context vector $\mathbf{c}_i$, a weighted sum of annotations of all source words. The weights come from scores computed by the alignment model, a one-hidden-layer multilayer perceptron of the previous decoder state and each annotation, normalized with a softmax.
• Soft alignment trained end to end. In phrase-based translation the alignment is a latent variable. Here it is computed directly as a soft alignment, so the gradient of the translation cost flows through it and the alignment model is trained jointly with the rest of the system by backpropagation. The authors read $\alpha_{ij}$ as the probability that target word $i$ is aligned to, or translated from, source word $j$, and $\mathbf{c}_i$ as an expected annotation.
• Bidirectional annotations. A bidirectional recurrent network reads the sentence both ways, so the annotation of each word summarizes the words before and after it, with a focus on its neighbourhood.
• Evidence against the bottleneck. The new model, RNNsearch, was compared with an RNN Encoder–Decoder trained on the same data with the same procedure, by sentence length, and the learned alignments were plotted as matrices.

The paper’s title and abstract speak of aligning and of a “(soft-)search”; the word attention occurs only in the last paragraph of Section 3.1, where the authors say that the weights intuitively implement a mechanism of attention in the decoder, which decides which parts of the source to pay attention to. Other papers used the word for it from the start. Sutskever et al.’s NIPS 2014 paper describes Bahdanau et al.’s model as using an attention mechanism and calls it an elegant variant of the differentiable attention of Graves (2013). Bahdanau et al. themselves discuss Graves’s handwriting synthesis model as the closest earlier alignment approach: its window over the characters was a mixture of Gaussians whose location could only move forward, whereas translation needs the weights to move in either direction.

Method

Section 3 defines the framework, and Appendix A gives the architecture used in the experiments. The scoring and weighting equations, $e_{ij} = a(\mathbf{s}_{i-1}, \mathbf{h}_j)$ followed by a softmax over $j$ and the weighted sum of the annotations (Eqs. 5–6), are written out in the attention mechanism article together with the additive score $\mathbf{v}_a^\top \tanh(W_a \mathbf{s}_{i-1} + U_a \mathbf{h}_j)$ of Appendix A.1.2. What follows is the rest of the model as published (v7, the final arXiv version).

Encoder. A forward and a backward recurrent network, each with $n = 1000$ gated hidden units, read the source words through a shared 620-dimensional word embedding. The annotation of word $j$ is the concatenation $\mathbf{h}_j = [\overrightarrow{\mathbf{h}}_j; \overleftarrow{\mathbf{h}}_j]$, a 2,000-dimensional vector (Eq. 7).

Decoder. The decoder is a gated recurrent network with $n = 1000$ units whose candidate state and both gates receive the context vector (Appendix A.2.2, biases omitted as in the paper):

$$\begin{aligned} \mathbf{s}_i &= (1 - \mathbf{z}_i) \circ \mathbf{s}_{i-1} + \mathbf{z}_i \circ \tilde{\mathbf{s}}_i, \\ \tilde{\mathbf{s}}_i &= \tanh\big(W E y_{i-1} + U [\mathbf{r}_i \circ \mathbf{s}_{i-1}] + C \mathbf{c}_i\big), \\ \mathbf{z}_i &= \sigma\big(W_z E y_{i-1} + U_z \mathbf{s}_{i-1} + C_z \mathbf{c}_i\big), \qquad \mathbf{r}_i = \sigma\big(W_r E y_{i-1} + U_r \mathbf{s}_{i-1} + C_r \mathbf{c}_i\big), \end{aligned}$$

where $E y_{i-1}$ is the embedding of the previous target word and $\circ$ is element-wise multiplication. The initial state is $\mathbf{s}_0 = \tanh(W_s \overleftarrow{\mathbf{h}}_1)$, computed from the backward encoder’s state at the first word, which has read the whole sentence. The alignment model’s hidden layer also has $n' = 1000$ units. Because it is evaluated $T_x \times T_y$ times for a sentence pair, the authors kept it to a single layer, and they note that $U_a \mathbf{h}_j$ does not depend on $i$ and can be computed once per sentence.

Output. A deep output layer with one maxout hidden layer predicts the next word:

$$\tilde{\mathbf{t}}_i = U_o \mathbf{s}_{i-1} + V_o E y_{i-1} + C_o \mathbf{c}_i, \qquad t_{i,j} = \max\{\tilde{t}_{i,2j-1}, \tilde{t}_{i,2j}\}, \qquad p(y_i \mid \mathbf{s}_i, y_{i-1}, \mathbf{c}_i) \propto \exp(y_i^\top W_o \mathbf{t}_i),$$

with $l = 500$ maxout units, each the maximum of two linear units. The appendix notes that fixing $\mathbf{c}_i$ to the last forward state $\overrightarrow{\mathbf{h}}_{T_x}$ turns the model into the RNN Encoder–Decoder.

Training. Recurrent matrices were initialized as random orthogonal matrices, $W_a$ and $U_a$ from a Gaussian with standard deviation 0.001, $\mathbf{v}_a$ and all biases at zero, and every other matrix from a Gaussian with standard deviation 0.01 (Appendix B.1). The models were trained by stochastic gradient descent with Adadelta ($\epsilon = 10^{-6}$, $\rho = 0.95$) on minibatches of 80 sentences, rescaling the gradient whenever its L2 norm exceeded 1. Because each update took time proportional to the longest sentence in the minibatch, every 20 updates the implementation fetched 1,600 sentence pairs, sorted them by length and split them into 20 minibatches. Translations were decoded with beam search. The paper points to the authors’ GroundHog repository for the implementations and thanks the developers of Theano.

Training Setup

The data were the English–French parallel corpora of the WMT’14 translation task: Europarl (61 million words), news commentary (5.5 million), UN (421 million) and two crawled corpora of 90 and 272.5 million words, 850 million in all, reduced to 348 million words with the data-selection method of Axelrod et al. (2011), following Cho et al. No monolingual data were used. The development set was news-test-2012 and news-test-2013 together, and the test set news-test-2014, 3,003 sentences. Each language had a shortlist of its 30,000 most frequent words, all other words being mapped to a single unknown-word token.

Two architectures were each trained twice, on sentences of up to 30 and up to 50 words: the RNN Encoder–Decoder of Cho et al. with 1,000 hidden units in encoder and decoder (RNNencdec-30 and -50), and RNNsearch (RNNsearch-30 and -50). Section 4.2 says each model was trained for about five days. Table 2 gives 108 to 113 hours for the four main models, on NVIDIA TITAN BLACK GPUs for the -30 models and Quadro K-6000 GPUs for the -50 models. A fifth model, RNNsearch-50⋆, was trained much longer, until development performance stopped improving: 252 hours, 6.67 × 10⁵ updates and 5.0 epochs, against 2.88 × 10⁵ updates and 2.2 epochs for RNNsearch-50.

Results

Translation Quality

Table 1 reports BLEU on news-test-2014, a metric of n-gram overlap with the reference translation, on all sentences and on the sentences with no unknown word in the source or the reference, where the models were also not allowed to generate the unknown-word token. The values are identical in all seven versions.

Model	All sentences	No unknown words
RNNencdec-30	13.93	24.19
RNNsearch-30	21.50	31.44
RNNencdec-50	17.82	26.71
RNNsearch-50	26.75	34.16
RNNsearch-50⋆	28.45	36.15
Moses	33.30	35.63

RNNsearch beat the encoder–decoder in every comparison, and RNNsearch-30 even beat RNNencdec-50, which had seen longer training sentences. Against Moses, the conventional phrase-based system, the claim is narrower than the abstract’s “comparable” suggests. On the sentences without unknown words, the longer-trained RNNsearch-50⋆ scored 0.52 BLEU above Moses. The authors call RNNsearch’s parity with Moses on this subset a significant achievement, because Moses also used a separate 418-million-word monolingual corpus. On the full test set, Moses led RNNsearch-50⋆ by 4.85 BLEU; the paper does not say how many sentences the subset holds. For context from the same test set, Sutskever et al.’s NIPS paper lists Bahdanau et al.’s 28.45 beside its own results: 30.59 for a single reversed four-layer LSTM and 34.81 for an ensemble of five, with an 80,000-word target vocabulary.

Long Sentences

Figure 2 plots BLEU against sentence length on the full test set. Both encoder–decoders fall sharply as sentences get longer; RNNsearch-30 and RNNsearch-50 are more robust, and the authors report no deterioration for RNNsearch-50 even at 50 words or more. Section 5.2.2 gives two test sentences, of 40 and 33 words by our count. The encoder–decoder translates the first correctly up to the words “a medical centre” and then replaces the clause about the doctor’s status as a health care worker with “based on his state of health”; in the second it drifts after about 30 words and drops a closing quotation mark. The authors judge RNNsearch-50’s translations of both correct. Appendix C (Table 3) repeats these two sentences and adds a third, each with the reference, both models’ outputs and Google Translate’s, run on 27 August 2014.

Alignments

Figure 3 shows the weights $\alpha_{ij}$ of RNNsearch-50 as grayscale matrices, English source words along one axis and the generated French along the other, for an arbitrary sentence and three test sentences drawn at random among those of 10 to 20 words without unknown words. The alignments are mostly monotonic, with strong weights along the diagonal, but not always. In “The agreement on the European Economic Area was signed in August 1992”, the model aligns “zone” with “Area”, jumping over two words, and then looks back one word at a time to produce “zone économique européenne”. In “the man” translated as “l’homme”, the model attends to both English words when writing “l’”, which the authors use to argue for soft over hard alignment: the French article depends on the following noun. Soft weights also handle phrases of different lengths without mapping words to an empty token.

Versions

The arXiv record has seven versions. The dates are from the submission history; the changes are from our comparison of the texts.

Version	Date	Changes
v1	1 Sep 2014	First version, no venue header; Sutskever et al.’s paper cited with an anonymized title, being under review
v2	4 Sep 2014	Accents corrected in Google Translate outputs
v3	7 Oct 2014	Sutskever et al. cited by title; Forcada and Ñeco (1997) added to the reference list
v4	19 Dec 2014	“Under review” header for ICLR 2015; Forcada and Ñeco cited in the background; baseline renamed from RNNenc to RNNencdec in the text and Table 1; context terms added to the decoder gates in Appendix A.2 (A.2.2 from v5)
v5	22 Mar 2015	“Published as a conference paper at ICLR 2015”; architectural choices moved from Section 3.3 to Appendix A.1; second crawled corpus corrected from 780 to 272.5 million words
v6	24 Apr 2015	Funding acknowledgement for Bahdanau added; output layer computed from $\mathbf{s}_{i-1}$ instead of $\mathbf{s}_i$; decoder-state equations in Appendices A.1.1 and A.2.2 changed to take $y_i$
v7	19 May 2016	Decoder-state equations returned to $y_{i-1}$ throughout; missing matrix $W$ added to the decoder’s candidate state

The numbers in Tables 1 and 2 are the same in every version. The v1–v4 corpus sizes did not add up to the stated 850 million words, and the same 780 million appears in both of Cho et al.’s 2014 papers. The previous-word input took three versions to settle: in the gated-unit equations of v1 to v5 (Section 3.3.1 in v1–v4, Appendix A.1.1 in v5) the candidate state uses $e(y_{i-1})$ but the two gates use $e(y_i)$, while Appendix A.2 uses $y_{i-1}$ throughout; v6 changed both sections to $y_i$, and v7 made both $y_{i-1}$, which matches the decoder definition $\mathbf{s}_i = f(\mathbf{s}_{i-1}, y_{i-1}, \mathbf{c}_i)$ of Section 3.1. A few inconsistencies remain in v7: Figure 2 and Tables 2 and 3 still say RNNenc; Appendix A.1.2 gives the alignment model’s matrix sizes in $n$ and A.2.2 in $n'$, equal only because both are 1,000; the output probability is written as conditioned on $\mathbf{s}_i$ but computed from $\mathbf{s}_{i-1}$; and Table 2’s caption calls the negative log-likelihood the average conditional log-probability.

The ICLR 2015 pages link to the unversioned arXiv record rather than to a separate proceedings copy; v5 is the first to carry the “published” header. The conference, held in San Diego on 7–9 May 2015, listed the paper among the oral presentations and in the poster session, and its schedule places the oral on 9 May, directly after the morning keynote. Citations differ in year: Xu et al. (2015) and Vaswani et al. (2017), among others, cite it as a 2014 arXiv preprint.

Impact

In Translation

Within a year others refined the mechanism. Luong, Pham and Manning (EMNLP 2015) proposed global and local attention for stacked LSTM translators and a simpler computation path, in which the current decoder state rather than the previous one computes the weights; they noted that Bahdanau et al. had tried only one scoring function, the additive one, and found others better on their tasks. They also measured alignment quality against gold word alignments, which Bahdanau et al. had shown only for sample sentences. Vaswani et al. (2017) built the transformer from attention alone and described its encoder–decoder attention as mimicking that of earlier sequence-to-sequence models, citing this paper among them; they also cite it for additive attention, one of the two common attention functions they compare (see the Attention Is All You Need paper).

In April 2025 ICLR named the paper runner-up for its Test of Time Award, given to papers from ICLR 2015 judged to have had a lasting impact; the award went to Kingma and Ba’s Adam. The announcement credits the paper with introducing a form of attention and with changing how sequence-to-sequence models process information.

From Translation to Vision

The paper has no images in it, and its authors did not apply the model to vision. The step to vision came partly from the same lab. Xu et al.’s “Show, Attend and Tell” (arXiv, 10 February 2015; ICML 2015), with Cho and Bengio among its authors, generated image captions with an LSTM decoder that attends over the $14 \times 14 \times 512$ feature map of a VGG network, treated as 196 annotation vectors. The authors say their work directly extends Bahdanau et al.’s, and their deterministic “soft” attention is the weighted sum of annotations “as proposed by Bahdanau et al.”, with weights from a multilayer perceptron conditioned on the previous decoder state: the RNNsearch mechanism with image regions in place of source words. They added a stochastic “hard” variant that samples one location. Among the works Xu et al. extend is also the glimpse model of Mnih et al. (2014), which had used attention in vision about two months before Bahdanau et al.’s preprint, but chose where to look with reinforcement learning rather than by a differentiable weighted sum. On COCO, soft attention scored 24.3 BLEU-4 and hard attention 25.0 (their Table 1). Captioning with attention, and what replaced it, is followed in the image captioning article.

The second route was the transformer. Its decoder-to-encoder attention is the same idea, a target position weighting all source positions, with dot-product scores and several heads; in vision it became the cross-attention through which DETR’s object queries read image features and text-to-image models read a prompt. Its self-attention applied to image patches became the Vision Transformer. In our reading the inheritance is in the form, soft and content-based weights over every input position trained by backpropagation, rather than in the additive score, which vision models mostly replaced with scaled dot products.

Limitations

• One task, one language pair. All experiments are English-to-French translation on one test set, with no vision, speech or other language pairs. About other tasks the paper says only that its cost may limit them.
• The cost of searching everywhere. The authors note that the model computes a weight for every source word at every target word; for translation, with most sentences of 15–40 words, they judged this acceptable, but they warned that it might limit the scheme on other tasks. For images, with hundreds or thousands of positions, the same count grows quickly.
• Rare words. Every word outside the 30,000-word shortlist becomes one unknown token, and the strongest comparison with Moses uses only sentences free of them. The conclusion names handling unknown and rare words as the main open problem.
• Uneven training budgets. The best result comes from a model trained more than twice as long as the others, with no matching encoder–decoder, and each encoder–decoder and its RNNsearch counterpart ran for similar hours on the same type of GPU but made different numbers of updates (8.46 against 4.71 × 10⁵ for the -30 pair). In our reading the comparison is at roughly equal wall-clock time, not at equal training.
• Qualitative alignments. The alignment matrices are four examples, read by eye; the paper does not evaluate alignment quality, and whether such weights explain a model’s output is disputed (see the attention mechanism article).
• An appendix that changed under readers. Readers of different versions find different equations for the gate inputs and the output layer, as the table above shows.

What Came After

By 2017 Vaswani et al. could describe attention mechanisms as an integral part of sequence modeling and transduction models, almost always used together with a recurrent network; the transformer kept the attention and removed the recurrence. Their argument for it, that short paths between any two positions make long-range dependencies easier to learn, is in our reading the same argument Bahdanau et al. made against the fixed-length vector. In vision, attention spread as soft attention over feature maps, channel and spatial gating, non-local blocks and finally attention-only backbones, a history traced in the attention mechanism article. In our reading, what lasted is less the paper’s model, a bidirectional gated recurrent encoder with additive attention, than its question: how a model should find, for each output, the part of its input it needs.