Plan, Attend, Generate: Character-level Neural Machine Translation with Planning in the Decoder

TL;DR

This paper introduces a planning-based mechanism into character-level neural machine translation, improving alignment quality and translation performance by explicitly modeling future alignments within an end-to-end trainable framework.

Contribution

It presents a novel planning mechanism for NMT decoders that explicitly models future alignments, enhancing translation quality and interpretability.

Findings

Outperforms strong baseline on WMT'15 corpus

Computes intuitive alignments

Achieves better performance with fewer parameters

Abstract

We investigate the integration of a planning mechanism into an encoder-decoder architecture with an explicit alignment for character-level machine translation. We develop a model that plans ahead when it computes alignments between the source and target sequences, constructing a matrix of proposed future alignments and a commitment vector that governs whether to follow or recompute the plan. This mechanism is inspired by the strategic attentive reader and writer (STRAW) model. Our proposed model is end-to-end trainable with fully differentiable operations. We show that it outperforms a strong baseline on three character-level decoder neural machine translation on WMT'15 corpus. Our analysis demonstrates that our model can compute qualitatively intuitive alignments and achieves superior performance with fewer parameters.

Figures

Tables

Table 1. Table 1: The results of different models on WMT’15 task on English to German, English to Czech and English to Finnish language pairs. We report BLEU scores of each model computed via the multi-blue.perl script. The best-score of each model for each language pair appears in bold-face. We use newstest2013 as our development set, newstest2014 as our "Test 2014" and newstest2015 as our "Test 2015" set. ( ) † \left({}^{\dagger}\right) denotes the results of the baseline that we trained using the hyperparameters reported in Chung et al. ( 2016 ) and the code provided with that paper. For our baseline, we only report the median result, and do not have multiple runs of our models.

	Model	Layer Norm	Dev	Test 2014	Test 2015
En$\to$De	Baseline	✗	21.57	21.33	23.45
	Baseline†	✗	21.4	21.16	22.1
	PAG	✗	21.52	21.35	22.21
		✓	22.12	21.93	22.83
	rPAG	✗	21.81	21.71	22.45
		✓	21.67	21.81	22.73
En$\to$Cs	Baseline	✗	17.68	19.27	16.98
	PAG	✗	17.44	18.72	16.99
		✓	18.78	20.9	18.59
	rPAG	✗	17.83	19.54	17.79
En$\to$Fi	Baseline	✗	11.19	-	10.93
	PAG	✗	11.51	-	11.13
		✓	12.67	-	11.84
	rPAG	✗	11.50	-	10.59

Table 2. Table 2: Randomly chosen example translations from the development-set.

	Groundtruth	Our Model (PAG + Biscale)	Baseline (Biscale)
1	Eine republikanische Strategie , um der Wiederwahl von Obama entgegenzutreten	Eine republikanische Strategie gegen die Wiederwahl von Obama	Eine republikanische Strategie zur Bekämpfung der Wahlen von Obama
2	Die Führungskräfte der Republikaner rechtfertigen ihre Politik mit der Notwendigkeit , den Wahlbetrug zu bekämpfen .	Republikanische Führungspersönlichkeiten haben ihre Politik durch die Notwendigkeit gerechtfertigt , Wahlbetrug zu bekämpfen .	Die politischen Führer der Republikaner haben ihre Politik durch die Notwendigkeit der Bekämpfung des Wahlbetrugs gerechtfertigt .
3	Der Generalanwalt der USA hat eingegriffen , um die umstrittensten Gesetze auszusetzen .	Die Generalstaatsanwälte der Vereinigten Staaten intervenieren , um die umstrittensten Gesetze auszusetzen .	Der Generalstaatsanwalt der Vereinigten Staaten hat dazu gebracht , die umstrittensten Gesetze auszusetzen .
4	Sie konnten die Schäden teilweise begrenzen	Sie konnten die Schaden teilweise begrenzen	Sie konnten den Schaden teilweise begrenzen .
5	Darüber hinaus haben Sie das Recht von Einzelpersonen und Gruppen beschränkt , jenen Wählern Hilfestellung zu leisten , die sich registrieren möchten .	Darüber hinaus begrenzten sie das Recht des Einzelnen und der Gruppen , den Wählern Unterstützung zu leisten , die sich registrieren möchten .	Darüber hinaus unterstreicht Herr Beaulieu die Bedeutung der Diskussion Ihrer Bedenken und Ihrer Familiengeschichte mit Ihrem Arzt .

Equations

$${\mathbf{s}}_t=f_{\text{dec-rnn}}({\mathbf{s}}_{t-1},{\mathbf{y}}_t,{\psi }_t).$$

$${\psi }_t=\sum _i^{|X|}{\alpha }_{ti}{\mathbf{h}}_i.$$

$${\bar{\mathbf{A}}}_t[i]=f_{\text{align}}({\mathbf{s}}_{t-1},{\mathbf{h}}_j,{\beta }_t^i,{\mathbf{y}}_t),$$

$${\mathbf{u}}_{ti}$$

$${\mathbf{A}}_t[:,i]$$

$${\mathbf{c}}_t$$

$${\bar{\mathbf{c}}}_t$$

$$p({\mathbf{y}}_t|{\mathbf{y}}_{<t},\mathbf{x})\propto {\mathbf{y}}_t^{\top }\exp ({\mathbf{W}}_o{f}_o({\mathbf{s}}_t,{\mathbf{y}}_{t-1},{\psi }_t)),$$

$$\mathcal{L}=-\frac{1}{N}\sum _{n=1}^N\log p_{\theta }({\mathbf{y}}^{(n)}|{\mathbf{x}}^{(n)}),$$

$${\mathcal{L}}_{\text{com}}={\lambda }_{\text{com}}\sum _{t=1}^{|X|}\sum _{i=0}^k||\frac{1}{k}-{\mathbf{c}}_{ti}|{|}_2^2,$$

Full text

1 Introduction

Character-level neural machine translation (NMT) is an attractive research problem (Lee et al., 2016; Chung et al., 2016; Luong and Manning, 2016) because it addresses important issues encountered in word-level NMT. Word-level NMT systems can suffer from problems with rare words(Gulcehre et al., 2016) or data sparsity, and the existence of compound words without explicit segmentation in certain language pairs can make learning alignments and translations more difficult. Character-level neural machine translation mitigates these issues.

In this work we propose to augment the encoder-decoder model for character-level NMT by integrating a planning mechanism. Specifically, we develop a model that uses planning to improve the alignment between input and output sequences. Our model’s encoder is a recurrent neural network (RNN) that reads the source (a sequence of byte pairs representing text in some language) and encodes it as a sequence of vector representations; the decoder is a second RNN that generates the target translation character-by-character in the target language. The decoder uses an attention mechanism to align its internal state to vectors in the source encoding. It creates an explicit plan of source-target alignments to use at future time-steps based on its current observation and a summary of its past actions. At each time-step it may follow or modify this plan. This enables the model to plan ahead rather than attending to what is relevant primarily at the current generation step. More concretely, we augment the decoder’s internal state with (i) an alignment plan matrix and (ii) a commitment plan vector. The alignment plan matrix is a template of alignments that the model intends to follow at future time-steps, i.e., a sequence of probability distributions over input tokens. The commitment plan vector governs whether to follow the alignment plan at the current step or to recompute it, and thus models discrete decisions. This planning mechanism is inspired by the strategic attentive reader and writer (STRAW) of Vezhnevets et al. (2016).

Our work is motivated by the intuition that, although natural language is output step-by-step because of constraints on the output process, it is not necessarily conceived and ordered according to only local, step-by-step interactions. Sentences are not conceived one word at a time. Planning, that is, choosing some goal along with candidate macro-actions to arrive at it, is one way to induce coherence in sequential outputs like language. Learning to generate long coherent sequences, or how to form alignments over long input contexts, is difficult for existing models. NMT performance of encoder-decoder models with attention deteriorates as sequence length increases  (Cho et al., 2014; Sutskever et al., 2014), and this effect can be more pronounced at the character-level NMT. This is because character sequences are longer than word sequences. A planning mechanism can make the decoder’s search for alignments more tractable and more scalable.

We evaluate our proposed model and report results on character-level translation tasks from WMT’15 for English to German, English to Finnish, and English to Czech language pairs. On almost all pairs we observe improvements over a baseline that represents the state of the art in neural character-level translation. In our NMT experiments, our model outperforms the baseline despite using significantly fewer parameters and converges faster in training.

2 Planning for Character-level Neural Machine Translation

We now describe how to integrate a planning mechanism into a sequence-to-sequence architecture with attention (Bahdanau et al., 2015). Our model first creates a plan, then computes a soft alignment based on the plan, and generates at each time-step in the decoder. We refer to our model as PAG (Plan-Attend-Generate).

2.1 Notation and Encoder

As input our model receives a sequence of tokens, $X=(x_{0},\cdots,x_{|X|})$, where $|X|$ denotes the length of $X$. It processes these with the encoder, a bidirectional RNN. At each input position $i$ we obtain annotation vector $\mathbf{h}_{i}$ by concatenating the forward and backward encoder states, $\mathbf{h}_{i}=[\mathbf{h}_{i}^{\rightarrow};\mathbf{h}_{i}^{\leftarrow}]$, where $\mathbf{h}_{i}^{\rightarrow}$ denotes the hidden state of the encoder’s forward RNN and $\mathbf{h}_{i}^{\leftarrow}$ denotes the hidden state of the encoder’s backward RNN.

Through the decoder the model predicts a sequence of output tokens, $Y=(y_{1},\cdots,y_{|Y|})$. We denote by $\mathbf{s}_{t}$ the hidden state of the decoder RNN generating the target output token at time-step $t$.

2.2 Alignment and Decoder

Our goal is a mechanism that plans which parts of the input sequence to focus on for the next $k$ time-steps of decoding. For this purpose, our model computes an alignment plan matrix $\mathbf{A}_{t}\in\mathbb{R}^{k\times|X|}$ and commitment plan vector $\mathbf{c}_{t}\in\mathbb{R}^{k}$ at each time-step. Matrix $\mathbf{A}_{t}$ stores the alignments for the current and the next $k-1$ timesteps; it is conditioned on the current input, i.e. the token predicted at the previous time-step $\mathbf{y}_{t}$, and the current context $\psi_{t}$, which is computed from the input annotations $\mathbf{h}_{i}$. The recurrent decoder function, $f_{\text{dec-rnn}}(\cdot)$, receives $\mathbf{s}_{t-1}$, $\mathbf{y}_{t}$, $\psi_{t}$ as inputs and computes the hidden state vector

[TABLE]

Context $\psi_{t}$ is obtained by a weighted sum of the encoder annotations,

[TABLE]

The alignment vector $\alpha_{t}=\mathtt{softmax}(\mathbf{A}_{t}[0])\in\mathbb{R}^{|X|}$ is a function of the first row of the alignment matrix. At each time-step, we compute a candidate alignment-plan matrix $\bar{\mathbf{A}}_{t}$ whose entry at the $i^{th}$ row is

[TABLE]

where $f_{\text{align}}(\cdot)$ is an MLP and $\beta^{i}_{t}$ denotes a summary of the alignment matrix’s $i^{th}$ row at time $t-1$. The summary is computed using an MLP, $f_{r}(\cdot)$, operating row-wise on $\mathbf{A}_{t-1}$: $\beta^{i}_{t}=f_{r}(\mathbf{A}_{t-1}[i])$.

The commitment plan vector $\mathbf{c}_{t}$ governs whether to follow the existing alignment plan, by shifting it forward from $t-1$, or to recompute it. Thus, $\mathbf{c}_{t}$ represents a discrete decision. For the model to operate discretely, we use the recently proposed Gumbel-Softmax trick (Jang et al., 2016; Maddison et al., 2016) in conjunction with the straight-through estimator  (Bengio et al., 2013) to backpropagate through $\mathbf{c}_{t}$.111We also experimented with training $\mathbf{c}_{t}$ using REINFORCE (Williams, 1992) but found that Gumbel-Softmax led to better performance. The model further learns the temperature for the Gumbel-Softmax as proposed in Gulcehre et al. (2017). Both the commitment vector and the action plan matrix are initialized with ones; this initialization is not modified through training.

Alignment-plan update

Our decoder updates its alignment plan as governed by the commitment plan. Denoted by $g_{t}$ the first element of the discretized commitment plan $\bar{\mathbf{c}}_{t}$. In more detail, $g_{t}=\bar{\mathbf{c}}_{t}[0]$, where the discretized commitment plan is obtained by setting $\mathbf{c}_{t}$’s largest element to 1 and all other elements to 0. Thus, $g_{t}$ is a binary indicator variable; we refer to it as the commitment switch. When $g_{t}=0$, the decoder simply advances the time index by shifting the action plan matrix $\mathbf{A}_{t-1}$ forward via the shift function $\rho(\cdot)$. When $g_{t}=1$, the controller reads the action-plan matrix to produce the summary of the plan, $\beta^{i}_{t}$. We then compute the updated alignment plan by interpolating the previous alignment plan matrix $\mathbf{A}_{t-1}$ with the candidate alignment plan matrix $\bar{\mathbf{A}}_{t}$. The mixing ratio is determined by a learned update gate $\mathbf{u}_{t}\in\mathbb{R}^{k\times|X|}$, whose elements $\mathbf{u}_{ti}$ correspond to tokens in the input sequence and are computed by an MLP with sigmoid activation, $f_{\text{up}}(\cdot)$:

[TABLE]

To reiterate, the model only updates its alignment plan when the current commitment switch $g_{t}$ is active. Otherwise it uses the alignments planned and committed at previous time-steps.

Commitment-plan update

The commitment plan also updates when $g_{t}$ becomes 1. If $g_{t}$ is 0, the shift function $\rho(\cdot)$ shifts the commitment vector forward and appends a [math]-element. If $g_{t}$ is 1, the model recomputes $\mathbf{c}_{t}$ using a single layer MLP ($f_{c}(\cdot)$) followed by a Gumbel-Softmax, and $\bar{\mathbf{c}}_{t}$ is recomputed by discretizing $\mathbf{c}_{t}$ as a one-hot vector:

[TABLE]

We provide pseudocode for the algorithm to compute the commitment plan vector and the action plan matrix in Algorithm 1. An overview of the model is depicted in Figure 1.

2.2.1 Alignment Repeat

In order to reduce the model’s computational cost, we also propose an alternative approach to computing the candidate alignment-plan matrix at every step. Specifically, we propose a model variant that reuses the alignment from the previous time-step until the commitment switch activates, at which time the model computes a new alignment. We call this variant repeat, plan, attend, and generate (rPAG). rPAG can be viewed as learning an explicit segmentation with an implicit planning mechanism in an unsupervised fashion. Repetition can reduce the computational complexity of the alignment mechanism drastically; it also eliminates the need for an explicit alignment-plan matrix, which reduces the model’s memory consumption as well. We provide pseudocode for rPAG in Algorithm 1.

2.3 Training

We use a deep output layer Pascanu et al. (2013) to compute the conditional distribution over output tokens,

[TABLE]

where $\mathbf{W}_{o}$ is a matrix of learned parameters and we have omitted the bias for brevity. Function $f_{o}$ is an MLP with $\tanh$ activation.

The full model, including both the encoder and decoder, is jointly trained to minimize the (conditional) negative log-likelihood

[TABLE]

where the training corpus is a set of $(\mathbf{x}^{(n)},\mathbf{y}^{(n)})$ pairs and ${\boldsymbol{\theta}}$ denotes the set of all tunable parameters. As noted in (Vezhnevets et al., 2016), the proposed model can learn to recompute very often which decreases the utility of planning. In order to avoid this behavior, we introduce a loss that penalizes the model for committing too often,

[TABLE]

where $\lambda_{\text{com}}$ is the commitment hyperparameter and $k$ is the timescale over which plans operate.

3 Experiments

In our NMT experiments we use byte pair encoding (BPE) (Sennrich et al., 2015) for the source sequence and character representation for the target, the same setup described in Chung et al. (2016). We also use the same preprocessing as in that work.222Our implementation is based on the code available at https://github.com/nyu-dl/dl4mt-cdec

We test our planning models against a baseline on the WMT’15 tasks for English to German (En$\rightarrow$De), English to Czech (En$\rightarrow$Cs), and English to Finnish (En$\rightarrow$Fi) language pairs. We present the experimental results in Table 1.

As a baseline we use the biscale GRU model of Chung et al. (2016), with the attention mechanisms in both the baseline and (r)PAG conditioned on both layers of the encoder’s biscale GRU ($\mathbf{h}^{1}$ and $\mathbf{h}^{2}$ – see (Chung et al., 2016) for more detail). Our implementation reproduces the results in the original paper to within a small margin.

Table 1 shows that our planning mechanism generally improves translation performance over the baseline. It does this with fewer updates and fewer parameters. We trained (r)PAG for 350K updates on the training set, while the baseline was trained for 680K updates. We used 600 units in (r)PAG’s encoder and decoder, while the baseline used 512 in the encoder and 1024 units in the decoder. In total our model has about 4M fewer parameters than the baseline. We tested all models with a beam size of 15.

As can be seen from Table 1, layer normalization (Ba et al., 2016) improves the performance of the PAG model significantly. However, according to our results on En$\rightarrow$De, layer norm affects the performance of rPAG only marginally. Thus, we decided not to train rPAG with layer norm on other language pairs.

In Figure 3, we show qualitatively that our model constructs smoother alignments. In contrast to (r)PAG, we see that the baseline decoder aligns the first few characters of each word that it generates to a byte in the source sequence; for the remaining characters it places the largest alignment weight on the final, empty token of the source sequence. This is because the baseline becomes confident of which word to generate after the first few characters, and generates the remainder of the word mainly by relying on language-model predictions. As illustrated by the learning curves in Figure 2, we observe further that (r)PAG converges faster with the help of its improved alignments.

4 Conclusions and Future Work

In this work, we addressed a fundamental issue in neural generation of long sequences by integrating planning into the alignment mechanism of sequence-to-sequence architectures on machine translation problem. We proposed two different planning mechanisms: PAG, which constructs explicit plans in the form of stored matrices, and rPAG, which plans implicitly and is computationally cheaper. The (r)PAG approach empirically improves alignments over long input sequences. In machine translation experiments, models with a planning mechanism outperforms a state-of-the-art baseline on almost all language pairs using fewer parameters. As a future work, we plan to test our planning mechanism at the outputs of the model and other sequence-to-sequence tasks as well.

Appendix A Qualitative Translations from both Models

In Table 2, we present example translations from our model and the baseline along with the ground-truth.

333These examples are randomly chosen from the first 100 examples of the development set. None of the authors of this paper can speak or understand German.