Improving Neural Language Models by Segmenting, Attending, and Predicting the Future

TL;DR

This paper introduces an unsupervised method to improve neural language models by automatically inducing phrase structures and aligning context with subsequent phrases, leading to better performance and structural understanding.

Contribution

It presents a novel unsupervised approach for phrase induction and context-phrase alignment that enhances language model performance without requiring linguistic annotations.

Findings

Achieved state-of-the-art 17.4 perplexity on Wikitext-103

Model learns phrase-level structural knowledge unsupervised

Outperformed several strong baseline models

Abstract

Common language models typically predict the next word given the context. In this work, we propose a method that improves language modeling by learning to align the given context and the following phrase. The model does not require any linguistic annotation of phrase segmentation. Instead, we define syntactic heights and phrase segmentation rules, enabling the model to automatically induce phrases, recognize their task-specific heads, and generate phrase embeddings in an unsupervised learning manner. Our method can easily be applied to language models with different network architectures since an independent module is used for phrase induction and context-phrase alignment, and no change is required in the underlying language modeling network. Experiments have shown that our model outperformed several strong baseline models on different data sets. We achieved a new state-of-the-art performance of 17.4 perplexity on the Wikitext-103 dataset. Additionally, visualizing the outputs of the phrase induction module showed that our model is able to learn approximate phrase-level structural knowledge without any annotation.

Tables

Table 1. Table 1: Experimental results on Penn Treebank dataset. Compared with the AWD-LSTM baseline models, our method reduced the perplexity on test set by 1.6.

Model	#Params	Dev PPL	Test PPL
Inan et al. (2016) – Tied Variational LSTM	24M	75.7	73.2
Zilly et al. (2017) – Recurrent Highway Networks	23M	67.9	65.7
Shen et al. (2017) – PRPN	-	-	62.0
Pham et al. (2018) – Efficient NAS	24M	60.8	58.6
Melis et al. (2017) – 4-layer skip LSTM (tied)	24M	60.9	58.3
Shen et al. (2018b) – ON-LSTM	25M	58.3	56.2
Liu et al. (2018) – Differentiable NAS	23M	58.3	56.1
Merity et al. (2017) – AWD-LSTM	24M	60.7	58.8
Merity et al. (2017) – AWD-LSTM + finetuning	24M	60.0	57.3
Ours – AWD-LSTM + Phrase Induction - NS	24M	61.0	58.6
Ours – AWD-LSTM + Phrase Induction - Attention	24M	60.2	58.0
Ours – AWD-LSTM + Phrase Induction	24M	59.6	57.5
Ours – AWD-LSTM + Phrase Induction + finetuning	24M	57.8	55.7
Dai et al. (2018) – Transformer-XL	24M	56.7	54.5
Yang et al. (2017) – AWD-LSTM-MoS + finetuning	22M	56.5	54.4

Table 2. Table 2: Experimental results on Wikitext-2 dataset.

Model	#Params	Dev PPL	Test PPL
Inan et al. (2016) – Variational LSTM (tied)	28M	92.3	87.7
Inan et al. (2016) – VLSTM + augmented loss	28M	91.5	87.0
Grave et al. (2016) – LSTM	-	-	99.3
Grave et al. (2016) – LSTM + Neural cache	-	-	68.9
Melis et al. (2017) – 1-Layer LSTM	24M	69.3	69.9
Melis et al. (2017) – 2-Layer Skip Conn. LSTM	24M	69.1	65.9
Merity et al. (2017) – AWD-LSTM + finetuning	33M	68.6	65.8
Ours – AWD-LSTM + Phrase Induction	33M	68.4	65.2
Ours – AWD-LSTM + Phrase Induction + finetuning	33M	66.9	64.1

Table 3. Table 3: Experimental results on Wikitext-103 dataset.

Model	#Params	Test PPL
Grave et al. (2016) – LSTM	-	48.7
Bai et al. (2018) – TCN	-	45.2
Dauphin et al. (2017) – GCNN-8	-	44.9
Grave et al. (2016) – LSTM + Neural cache	-	40.8
Dauphin et al. (2017) – GCNN-14	-	37.2
Merity et al. (2018) – 4-layer QRNN	151M	33.0
Rae et al. (2018) – LSTM + Hebbian + Cache	-	29.9
Dai et al. (2018) – Transformer-XL Standard	151M	24.0
Baevski and Auli (2018) – Adaptive input	247M	20.5
Dai et al. (2018) – Transformer-XL Large	257M	18.3
Ours – Transformer-XL Large + Phrase Induction	257M	17.4

Equations

$$d_i=W_d\cdot [x_{i-n},x_{i-n+1},\cdots ,x_i{]}^T+b_d$$

$$h_i=W_h\cdot ReLU(d_i)+b_h$$

$$h_j-h_i>0$$

$$h_j-h_{j+1}>0$$

$$[c_1,c_2,\cdots ,c_T]=RN{N}^1([x_1,x_2,\cdots ,x_T])$$

$$[y_1,y_2,\cdots ,y_T]=RN{N}^2([c_1,c_2,\cdots ,c_T])$$

$$h_i=TCN([x_{i-n},x_{i-n+1},\cdots ,x_i])$$

$$p_{psc}^1(x_j)=\frac{1}{2}\cdot (f^{HT}(h_j-h_i)+1)$$

$$p_{psc}^2(x_j)=\frac{1}{2}\cdot (f^{HT}(h_j-h_{j+1})+1)$$

$$f^{HT}(x)=\left\{\begin{array}[]{rcl}-1&&{x\leq-\frac{1}{a}}\\ a\cdot x&&{-\frac{1}{a}<x\leq\frac{1}{a}}\\ 1&&{x>\frac{1}{a}}\end{array}\right.$$

$$p^{ind}(x_j)=\prod _{k=1}^j\hat{p}(x_k)$$

$$\scriptstyle\hat{p}(x_{k})=\left\{\begin{array}[]{rcl}&1&{k\leq i+1}\\ &1-p_{psc}^{1}(x_{k-1})\cdot p_{psc}^{2}(x_{k-1})&{k>i+1}\end{array}\right.$$

$${\alpha }_i=\frac{h_i\cdot p^{ind}(x_i)+c}{{\sum }_j{h}_j\cdot p^{ind}(x_j)+c}$$

$$s=W\cdot \sum _i{\alpha }_i\cdot e_i$$

$$p(x_1,x_2,\cdots ,x_n)=p(x_1)\cdot \prod _{i}p(x_{i+1}|x_1^i)$$

$$p(x_{i+1})=Softmax(e_{i+1}^T\cdot y_i)$$

$$p(p{h}_i|x_1^i)=\sigma (c_i^T\cdot s_i)$$

$$l_i^{CPA}=1-\sigma (c_i^T\cdot s_i)+\frac{1}{n}\sum _{j=1}^n\sigma (c_i^T\cdot s_j^{neg})$$

$$l=l^{LM}+\gamma \cdot l^{CPA}$$

Full text

1 Introduction

Neural language models are typically trained by predicting the next word given a past context Bengio et al. (2003). However, natural sentences are not constructed as simple linear word sequences, as they usually contain complex syntactic information. For example, a subsequence of words can constitute a phrase, and two non-neighboring words can depend on each other. These properties make natural sentences more complex than simple linear sequences.

Most recent work on neural language modeling learns a model by encoding contexts and matching the context embeddings to the embedding of the next word Bengio et al. (2003); Merity et al. (2017); Melis et al. (2017). In this line of work, a given context is encoded with a neural network, for example a long short-term memory (LSTM; Hochreiter and Schmidhuber, 1997) network, and is represented with a distributed vector. The log-likelihood of predicting a word is computed by calculating the inner product between the word embedding and the context embedding. Although most models do not explicitly consider syntax, they still achieve state-of-the-art performance on different corpora. Efforts have also been made to utilize structural information to learn better language models. For instance, parsing-reading-predict networks (PRPN; Shen et al., 2017) explicitly learn a constituent parsing structure of a sentence and predict the next word considering the internal structure of the given context with an attention mechanism. Experiments have shown that the model is able to capture some syntactic information.

Similar to word representation learning models that learns to match word-to-word relation matrices Mikolov et al. (2013); Pennington et al. (2014), standard language models are trained to factorize context-to-word relation matrices Yang et al. (2017). In such work, the context comprises all previous words observed by a model for predicting the next word. However, we believe that context-to-word relation matrices are not sufficient for describing how natural sentences are constructed. We argue that natural sentences are generated at a higher level before being decoded to words. Hence a language model should be able to predict the following sequence of words given a context. In this work, we propose a model that factorizes a context-to-phrase mutual information matrix to learn better language models. The context-to-phrase mutual information matrix describes the relation among contexts and the probabilities of phrases following given contexts. We make the following contributions in this paper:

• •

We propose a phrase prediction model that improves the performance of state-of-the-art word-level language models.

• •

Our model learns to predict approximate phrases and headwords without any annotation.

2 Related Work

Neural networks have been widely applied in natural language modeling and generation Bengio et al. (2003); Bahdanau et al. (2014) for both encoding and decoding. Among different neural architectures, the most popular models are recurrent neural networks (RNNs; Mikolov et al., 2010), long short-term memory networks (LSTMs; Hochreiter and Schmidhuber, 1997), and convolutional neural networks (CNNs; Bai et al., 2018; Dauphin et al., 2017).

Many modifications of network structures have been made based on these architectures. LSTMs with self-attention can improve the performance of language modeling Tran et al. (2016); Cheng et al. (2016). As an extension of simple self-attention, transformers Vaswani et al. (2017) apply multi-head self-attention and have achieved competitive performance compared with recurrent neural language models. A current state-of-the-art model, Transformer-XL Dai et al. (2018), applied both a recurrent architecture and a multi-head attention mechanism. To improve the quality of input word embeddings, character-level information is also considered Kim et al. (2016). It has also been shown that context encoders can learn syntactic information Shen et al. (2017).

However, instead of introducing architectural changes, for example a self-attention mechanism or character-level information, previous studies have shown that careful hyper-parameter tuning and regularization techniques on standard LSTM language models can obtain significant improvements Melis et al. (2017); Merity et al. (2017). Similarly, applying more careful dropout strategies can also improve the language models Gal and Ghahramani (2016); Melis et al. (2018). LSTM language models can be improved with these approaches because LSTMs suffer from serious over-fitting problems.

Recently, researchers have also attempted to improve language models at the decoding phase. Inan et al. (2016) showed that reusing the input word embeddings in the decoder can reduce the perplexity of language models. Yang et al. (2017) showed the low-rank issue in factorizing the context-to-word mutual information matrix and proposed a multi-head softmax decoder to solve the problem. Instead of predicting the next word by using only similarities between contexts and words, the neural cache model Grave et al. (2016) can significantly improve language modeling by considering the global word distributions conditioned on the same contexts in other parts of the corpus.

To learn the grammar and syntax in natural languages, Dyer et al. (2016) proposed the recurrent neural network grammar (RNNG) that models language incorporating a transition parsing model. Syntax annotations are required in this model. To utilize the constituent structures in language modeling without syntax annotation, parse-read-predict networks (PRPNs; Shen et al., 2017) calculate syntactic distances among words and computes self-attentions. Syntactic distances have been proved effective in constituent parsing tasks Shen et al. (2018a). In this work, we learn phrase segmentation with a model based on this method and our model does not require syntax annotation.

3 Syntactic Height and Phrase Induction

In this work, we propose a language model that not only predicts the next word of a given context, but also attempts to match the embedding of the next phrase. The first step of this approach is conducting phrase induction based on syntactic heights. In this section, we explain the definition of syntactic height in our approach and describe the basics ideas about whether a word can be included in an induced phrase.

Intuitively, the syntactic height of a word aims to capture its distance to the root node in a dependency tree. In Figure 1, the syntactic heights are represented by the red bars. A word has high syntactic height if it has low distance to the root node.

A similar idea, named syntactic distance, is proposed by Shen et al. (2017) for constructing constituent parsing trees. We apply the method for calculating syntactic distance to calculate syntactic height. Given a sequence of embeddings of input words $[x_{1},x_{2},\cdots,x_{n}]$, we calculate their syntactic heights with a temporal convolutional network (TCN) Bai et al. (2018).

[TABLE]

[TABLE]

where $h_{i}$ stands for the syntactic height of word $x_{i}$. The syntactic height $h_{i}$ for each word is a scalar, and $W_{h}$ is a $1\times D$ matrix, where $D$ is the dimensionality of $d_{i}$. These heights are learned and not imposed by external syntactic supervision. In Shen et al. (2017), the syntactic heights are used to generate context embeddings. In our work, we use the syntactic heights to predict induced phrases and calculate their embeddings.

We define the phrase induced by a word based on the syntactic heights. Consider two words $x_{i}$ and $x_{k}$. $x_{k}$ belongs to the phrase induced by $x_{i}$ if and only if for any $j\in(i,k)$, $h_{j}<max(h_{i},h_{k})$. For example, in Figure 1, the phrase induced by the red marked word the is “the morning flights”, since the syntactic height of the word morning, $h_{morning}<h_{flights}$. However, the word “to” does not belong to the phrase because $h_{flights}$ is higher than both $h_{the}$ and $h_{to}$. The induced phrase and the inducing dependency connection are labeled in blue in the figure.

Note that this definition of an induced phrase does not necessarily correspond to a phrase in the syntactic constituency sense. For instance, the words “to Houston” would be included in the phrase “the morning flights to Houston” in a traditional syntactic tree. Given the definition of induced phrases, we propose phrase segmenting conditions (PSCs) to find the last word of an induced phrase. Considering the induced phrase of the $i$-th word, $s_{i}=[x_{i},x_{i+1},\cdots,x_{j}]$. If $x_{j}$ is not the last word of a given sentence, there are two conditions that $x_{j}$ should satisfy:

(PSC-1) The syntactic height of $x_{j}$ must be higher than the height of $x_{i}$, that is

[TABLE]

• 2.

(PSC-2) The syntactic height of $x_{j+1}$ should be lower that $x_{j}$.

[TABLE]

Given the PSCs, we can decide the induced phrases for the sentence shown in Figure 1. The last word of the phrase induced by “United” is “canceled”, and the last word of the phrase induced by “flights” is “Houston”. For the word assigned the highest syntactic height, its induced phrase is all remaining words in the sentence.

4 Model

In this work, we formulate multi-layer neural language models as a two-part framework. For example, in a two-layer LSTM language model Merity et al. (2017), we use the first layer as phrase generator and the last layer as a word generator:

[TABLE]

[TABLE]

For a $L$-layer network, we can regard the first $L_{1}$ layers as the phrase generator and the next $L_{2}=L-L_{1}$ layers as the word generator. Note that we use $y_{i}$ to represent the hidden state output by the second layer instead of $h_{i}$, since $h_{i}$ in our work is defined as the syntactic height of $x_{i}$. In the traditional setting, the first layer does not explicitly learn the semantics of the following phrase because there is no extra objective function for phrase learning.

In this work, we force the first layer to output context embeddings $c_{i}$ for phrase prediction with three steps. Firstly, we predict the induced phrase for each word according to the PSCs proposed in Section 3. Secondly, we calculate the embedding of each phrase with a head-finding attention. Lastly, we align the context embedding and phrase embedding with negative sampling. The word generation is trained in the same way as standard language models. The diagram of the model is shown in Figure 2. The three steps are described next.

4.1 Phrase Segmentation

We calculate the syntactic height and predict the induced phrase for each word:

[TABLE]

where $TCN(\cdot)$ stands for the TCN model described in Equations (1) and (2), and $n$ is the width of the convolution window.

Based on the proposed phrase segmenting conditions (PSCs) described in the previous section, we predict the probability of a word being the first word outside a induced phrase. Firstly, we decide if each word, $x_{j-1},j\in(i+1,n]$, satisfies the two phrase segmenting conditions, PSC-1 and PSC-2. The probability that $x_{j}$ satisfies PSC-1 is

[TABLE]

Similarly, the probability that $x_{j}$ satisfies PSC-2 is

[TABLE]

where $f_{HT}$ stands for the HardTanh function with a temperature $a$:

[TABLE]

This approach is inspired by the context attention method proposed in the PRPN model Shen et al. (2017).

Then we can infer the probability of whether a word belongs to the induced phrase of $x_{i}$ with

[TABLE]

where $p^{ind}(x_{i})$ stands for the probability that $x_{i}$ belongs to the induced phrase, and

[TABLE]

Note that the factorization in Equation 10 assumes that words are independently likely to be included in the induced phrase of $x_{i}$.

4.2 Phrase Embedding with Attention

Given induced phrases, we can calculate their embeddings based on syntactic heights. To calculate the embedding of phrase $s=[x_{1},x_{2},\cdots,x_{n}]$, we calculate an attention distribution over the phrase:

[TABLE]

where $h_{i}$ stands for the syntactic height for word $x_{i}$ and $c$ is a constant real number for smoothing the attention distribution. Then we generate the phrase embedding with a linear transformation:

[TABLE]

where $e_{i}$ is the word embedding of $x_{i}$. In training, we apply a dropout layer on $s$.

4.3 Phrase and Word Prediction

A traditional language model learns the probability of a sequence of words:

[TABLE]

where $x_{1}^{i}$ stands for $x_{1},x_{2},\cdots,x_{i}$, which is the context used for predicting the next word, $x_{i+1}$. In most related studies, the probability $p(x_{i+1}|x_{1}^{i})$ is calculated with the output of the top layer of a neural network $y_{i}$ and the word representations $e_{i+1}$ learned by the decoding layer:

[TABLE]

The state-of-the-art neural language models contain multiple layers. The outputs of different hidden layers capture different level of semantics of the context. In this work, we force one of the hidden layers to align its output with the embeddings of induced phrases $s_{i}$. We apply an embedding model similar to Mikolov et al. (2013) to train the hidden output and phrase embedding alignment. We define the context-phrase alignment model as follows.

We first define the probability that a phrase $ph_{i}$ can be induced by context $[x_{1},\dots,x_{i}]$.

[TABLE]

where $\sigma(x)=\frac{1}{1+e^{-x}}$, and $c_{i}$ stands for the context embedding of $x_{1},x_{2},\cdots,x_{i}$ output by a hidden layer, defined in Equation 5. $s_{i}$ is the generated embedding of an induced phrase. The probability that a phrase $ph_{i}$ cannot be induced by context $[x_{1},\dots,x_{i}]$ is $1-p(ph_{i}|x_{1}^{i})$. This approach follows the method for learning word embeddings proposed in Mikolov et al. (2013).

We use an extra objective function and the negative sampling strategy to align context representations and the embeddings of induced phrases. Given the context embedding $c_{i}$, the induced phrase embedding $s_{i}$, and random sampled negative phrase embeddings $s_{i}^{neg}$, we train the neural network to maximize the likelihood of true induced phrases and minimize the likelihood of negative samples. we define the following objective function for context $i$:

[TABLE]

where $n$ stands for the number of negative samples. With this loss function, the model learns to maximize the similarity between the context and true induced phrase embeddings, and minimize the similarity between the context and negative samples randomly selected from the induced phrases of other words. In practice, this loss function is used as a regularization term with a coefficient $\gamma$:

[TABLE]

It worth noting that our approach is model-agnostic and and can be applied to various architectures. The TCN network for calculating the syntactic heights and phrase inducing is an independent module. In context-phrase alignment training with negative sampling, the objective function provides phrase-aware gradients and does not change the word-by-word generation process of the language model.

5 Experiments

We evaluate our model with word-level language modeling tasks on Penn Treebank (PTB; Mikolov et al., 2010), Wikitext-2 (WT2; Bradbury et al., 2016), and Wikitext-103 (WT103; Merity et al., 2016) corpora.

The PTB dataset has a vocabulary size of 10,000 unique words. The entire corpus includes roughly 40,000 sentences in the training set, and more than 3,000 sentences in both valid and test set.

The WT2 data is about two times larger the the PTB dataset. The dataset consists of Wikipedia articles. The corpus includes 30,000 unique words in its vocabulary and is not cleaned as heavily as the PTB corpus.

The WT103 corpus contains a larger vocabulary and more articles than WT2. It consists of 28k articles and more than 100M words in the training set. WT2 and WT103 corpora can evaluate the ability of capturing long-term dependencies Dai et al. (2018).

In each corpus, we apply our approach to publicly-available, state-of-the-art models. This demonstrates that our approach can improve different existing architectures. Our trained models will be published for downloading. The implementation of our models is publicly available.111https://github.com/luohongyin/PILM

5.1 Penn Treebank

We train a 3-layer AWD-LSTM language model Merity et al. (2017) on PTB data set. We use 1,150 as the number of hidden neurons and 400 as the size of word embeddings. We also apply the word embedding tying strategy Inan et al. (2016). We apply variational dropout for hidden states Gal and Ghahramani (2016) and the dropout rate is 0.25. We also apply weight dropout Merity et al. (2017) and set weight dropout rate as 0.5. We apply stochastic gradient descent (SGD) and averaged SGD (ASGD; Polyak and Juditsky, 1992) for training. The learning rate is 30 and we clip the gradients with a norm of 0.25. For the phrase induction model, we randomly sample 1 negative sample for each context, and the context-phrase alignment loss is given a coefficient of 0.5. The output of the second layer of the neural network is used for learning context-phrase alignment, and the final layer is used for word generation.

We compare the word-level perplexity of our model with other state-of-the-art models and our baseline is AWD-LSTM Merity et al. (2017). The experimental results are shown in Table 1. Although not as good as the Transformer-XL model Dai et al. (2018) and the mixture of softmax model Yang et al. (2017), our model significantly improved the AWD-LSTM, reducing 2.2 points of perplexity on the validation set and 1.6 points of perplexity on the test set. Note that the “finetuning” process stands for further training the language models with ASGD algorithm Merity et al. (2017).

We also did an ablation study without either headword attention or negative sampling (NS). The results are listed in Table 1. By simply averaging word vectors in the induced phrase Without the attention mechanism, the model performs worse than the full model by 0.5 perplexity, but is still better than our baseline, the AWD-LSTM model. In the experiment without negative sampling, we only use the embedding of true induced phrases to align with the context embedding. It is also indicated that the negative sampling strategy can improve the performance by 1.1 perplexity. Hence we just test the full model in the following experiments.

5.2 Wikitext-2

We also trained a 3-layer AWD-LSTM language model on the WT2 dataset. The network has the same input size, output size, and hidden size as the model we applied on PTB dataset, following the experiments done by Merity et al. (2017). Some hyper-parameters are different from the PTB language model. We use a batch size of 60. The embedding dropout rate is 0.65 and the dropout rate of hidden outputs is set to 0.2. Other hyper-parameters are the same as we set in training on the PTB dataset.

The experimental results are shown in Table 2. Our model improves the AWD-LSTM model by reducing 1.7 points of perplexity on both the validation and test sets, while we did not make any change to the architecture of the AWD-LSTM language model.

5.3 Wikitext-103

The current state-of-the-art language model trained on Wikitext-103 dataset is the Transformer-XL Dai et al. (2018). We apply our method on the state-of-the-art Transformer-XL Large model, which has 18 layers and 257M parameters. The input size and hidden size are 1024. 16 attention heads are used. We regard the first 14 layers as the phrase generator and the last 4 layers as the word generator. In other words, the context-phrase alignment is trained with the outputs of the 14th layer.

The model is trained on 4 Titan X Pascal GPUs, each of which has 12G memory. Because of the limitation of computational resources, we use our approach to fine-tune the officially released pre-trained Transformer-XL Large model for 1 epoch. The experimental results are shown in Table 3. Our approach got 17.4 perplexity with the officially released evaluation scripts, significantly outperforming all baselines and achieving new state-of-the-art performance222We did not show Dev PPLs in Table 3 since only the correct approach to reproduce the test PPL was provided with the pretrained Transformer-XL model..

6 Discussion

In this section, we show what is learned by training language models with the context-phrase alignment objective function by visualizing the syntactic heights output by the TCN model and the phrases induced by each target word in a sentence. We also visualize the headword attentions over the induced phrase.

The first example is the sentence showed in Figure 1. The sentence came from Jurafsky and Martin (2014) and did not appear in our training set. Figure 1 shows the syntactic heights and the induced phrase of “the” according to the ground-truth dependency information. Our model is not given such high-quality inputs in either training or evaluation.

Figure 3 visualizes the structure learned by our phrase induction model. The inferred syntactic heights are shown in Figure 3(a). Heights assigned to words “the” and “to” are significantly lower than others, while the verb “canceled” is assigned the highest in the sentence. Induced phrases are shown in Figure 3(b). The words at the beginning of each row stand for the target word of each step. Values in the matrix stand for attention weights for calculating phrase embedding. The weights are calculated with the phrase segmenting conditions (PSC) and the syntactic heights described in Equations 8 to 11. For the target word “united”, $h_{united}<h_{canceled}$ and $h_{canceled}>h_{the}$, hence the induced phrase of “united” is a single word “canceled”, and the headword attention of “canceled” is 1, which is indicated in the first row of Figure 3(b). The phrase induced by “canceled” is the entire following sequence, “the morning flights to houston”, since no following word has a higher syntactic height than the target word. It is also shown that the headword of the induced phrase of “canceled” is “flights”, which agrees with the dependency structure indicated in Figure 1.

More examples are shown in Figure 4. Figures 4(a) to 4(d) show random examples without any unknown word, while the examples shown in Figures 4(e) and 4(f) are randomly selected from sentences with unknown words, which are marked with the UNK symbol. The examples show that the phrase induction model does not always predict the exact structure represented by the dependency tree. For example, in Figure 4(b), the TCN model assigned the highest syntactic height to the word “market” and induced the phrase “expect a rough market” for the context “the fund managers”. However, in a ground-truth dependency tree, the verb “expect” is the word directly connected to the root node and therefore has the highest syntactic height.

Although not exactly matching linguistic dependency structures, the phrase-level structure predictions are reasonable. The segmentation is interpretable and the predicted headwords are appropriate. In Figure 4(c), the headwords are “trying”, “quality”, and “involvement”. The model is also robust with unknown words. In Figure 4(e), “the $<$unk$>$ council” is segmented as the induced phrase of “but a majority of”. In this case, the model recognized that the unknown word is dependent on “council”.

The sentence in Figure 4(f) includes even more unknown words. However, the model still correctly predicted the root word, the verb “speak”. For the target word “with”, the induced phrase is “strong $<$unk$>$”. Two unknown words are located in the last few words of the sentence. The model failed to induce the phrase “$<$unk$>$ and $<$unk$>$” for the word “do”, but still successfully split “$<$unk$>$” and “and”. Meanwhile, the attentions over the phrases induced by “speak”, “do”, and the first “$<$unk$>$” are not quite informative, suggesting that unknown words made some difficulties for headword prediction in this example. However, the unknown words are assigned significantly higher syntactic heights than the word “and”.

7 Conclusion

In this work, we improved state-of-the-art language models by aligning context and induced phrases. We defined syntactic heights and phrase segmentation rules. The model generates phrase embeddings with headword attentions. We improved the AWD-LSTM and Transformer-XL language models on different data sets and achieved state-of-the-art performance on the Wikitext-103 corpus. Experiments showed that our model successfully learned approximate phrase-level knowledge, including segmentation and headwords, without any annotation. In future work, we aim to capture better structural information and possible connections to unsupervised grammar induction.