Left-to-Right Dependency Parsing with Pointer Networks

TL;DR

This paper introduces a simpler, faster left-to-right dependency parser using pointer networks, achieving state-of-the-art accuracy on English and improvements across multiple languages.

Contribution

It presents a novel left-to-right transition-based dependency parsing algorithm that reduces transition steps and computational complexity using pointer networks.

Findings

Achieves 96.04% UAS and 94.43% LAS on English PTB.

Runs twice as fast as previous top-down parsers.

Improves accuracy in most tested languages.

Abstract

We propose a novel transition-based algorithm that straightforwardly parses sentences from left to right by building $n$ attachments, with $n$ being the length of the input sentence. Similarly to the recent stack-pointer parser by Ma et al. (2018), we use the pointer network framework that, given a word, can directly point to a position from the sentence. However, our left-to-right approach is simpler than the original top-down stack-pointer parser (not requiring a stack) and reduces transition sequence length in half, from 2$n$-1 actions to $n$. This results in a quadratic non-projective parser that runs twice as fast as the original while achieving the best accuracy to date on the English PTB dataset (96.04% UAS, 94.43% LAS) among fully-supervised single-model dependency parsers, and improves over the former top-down transition system in the majority of languages tested.

Tables

Table 1. Table 1: Accuracy comparison of state-of-the-art fully-supervised single-model dependency parsers on PT-SD. The first block contains transition-based algorithms and the second one, graph-based models. Systems marked with ∗ , including the improved variant described in Ma et al. ( 2018 ) of the graph-based parser by Dozat and Manning ( 2016 ) , are implemented under the same framework as our approach and use the same training settings. Like Ma et al. ( 2018 ) , we report the average accuracy over 5 repetitions.

Parser	UAS	LAS
Chen and Manning (2014)	91.8	89.6
Dyer et al. (2015)	93.1	90.9
Weiss et al. (2015)	93.99	92.05
Ballesteros et al. (2016)	93.56	91.42
Kiperwasser and Goldberg (2016)	93.9	91.9
Alberti et al. (2015)	94.23	92.36
Qi and Manning (2017)	94.3	92.2
Fernández-G and Gómez-R (2018)	94.5	92.4
Andor et al. (2016)	94.61	92.79
Ma et al. (2018)∗	95.87	94.19
This work∗	96.04	94.43
Kiperwasser and Goldberg (2016)	93.1	91.0
Wang and Chang (2016)	94.08	91.82
Cheng et al. (2016)	94.10	91.49
Kuncoro et al. (2016)	94.26	92.06
Zhang et al. (2017)	94.30	91.95
Ma and Hovy (2017)	94.88	92.96
Dozat and Manning (2016)	95.74	94.08
Ma et al. (2018)∗	95.84	94.21

Table 2. Table 2: Parsing accuracy of the top-down and left-to-right pointer-network-based parsers on test datasets of twelve languages from UD. Best results for each language are shown in bold and, apart from the average UAS and LAS, we also report the corresponding standard deviation over 3 runs.

	Top-down		Left-to-right
	UAS	LAS	UAS	LAS
bu	94.42$\pm$0.02	90.70$\pm$0.04	94.28$\pm$0.06	90.66$\pm$0.11
ca	93.83$\pm$0.02	91.96$\pm$0.01	94.07$\pm$0.06	92.26$\pm$0.05
cs	93.97$\pm$0.02	91.23$\pm$0.03	94.19$\pm$0.04	91.45$\pm$0.05
de	87.28$\pm$0.07	82.99$\pm$0.07	87.06$\pm$0.05	82.63$\pm$0.01
en	90.86$\pm$0.15	88.92$\pm$0.19	90.93$\pm$0.11	88.99$\pm$0.11
es	93.09$\pm$0.05	91.11$\pm$0.03	93.23$\pm$0.03	91.28$\pm$0.02
fr	90.97$\pm$0.09	88.22$\pm$0.12	90.90$\pm$0.04	88.14$\pm$0.10
it	94.08$\pm$0.04	92.24$\pm$0.06	94.28$\pm$0.06	92.48$\pm$0.02
nl	93.23$\pm$0.09	90.67$\pm$0.07	93.13$\pm$0.07	90.74$\pm$0.08
no	95.02$\pm$0.05	93.75$\pm$0.05	95.23$\pm$0.06	93.99$\pm$0.07
ro	91.44$\pm$0.11	85.80$\pm$0.14	91.58$\pm$0.08	86.00$\pm$0.07
ru	94.43$\pm$0.01	93.08$\pm$0.03	94.71$\pm$0.07	93.38$\pm$0.09

Equations

$$v_i^t=score(d_t,s_i)$$

$$a^t=softmax(v^t)$$

$$P_{\theta }(y|x)$$

$$=\prod _{i=1}^k\prod _{j=1}^{l_i}{P}_{\theta }(w_{i,j}|w_{i,<j},p_{<i},x)$$

$$P_{\theta }(y|x)$$

$$=\prod _{i=1}^n{P}_{\theta }(w_h|w_i,l_{<i},x)$$

Full text

1 Introduction

Dependency parsing, the task of automatically obtaining the grammatical structure of a sentence expressed as a dependency tree, has been widely studied by natural language processing (NLP) researchers in the last decades. Most of the models providing competitive accuracies fall into two broad families of approaches: graph-based McDonald et al. (2005a, b) and transition-based Yamada and Matsumoto (2003); Nivre (2003) dependency parsers.

Given an input sentence, a graph-based parser scores trees by decomposing them into factors, and performs a search for the highest-scoring tree.

In the past two years, this kind of dependency parsers have been ahead in terms of accuracy thanks to the graph-based neural architecture developed by Dozat and Manning (2016), which not only achieved state-of-the-art accuracies on the Stanford Dependencies conversion of the English Penn Treebank (hereinafter, PTB-SD), but also obtained the best results in the majority of languages in the CoNLL 2017 Shared Task Dozat et al. (2017). This tendency recently changed, since a transition-based parser developed by Ma et al. (2018) managed to outperform the best graph-based model in the majority of datasets tested.

Transition-based parsers incrementally build a dependency graph for an input sentence by applying a sequence of transitions. This results in more efficient parsers with linear time complexity for parsing projective sentences, or quadratic for handling non-projective structures, when implemented with greedy or beam search. However, their main weakness is the lack of access to global context information when transitions are greedily chosen. This favours error propagation, mainly affecting long dependencies that require a larger number of transitions to be built McDonald and Nivre (2011).

Many attempts have been made to alleviate the impact of error propagation in transition-based dependency parsing, but the latest and most successful approach was developed by Ma et al. (2018). In particular, they make use of pointer networks Vinyals et al. (2015) to implement a new neural network architecture called stack-pointer network. The proposed framework provides a global view of the input sentence by capturing information from the whole sentence and all the arcs previously built, crucial for reducing the effect of error propagation; and, thanks to an attention mechanism Bahdanau et al. (2014); Luong et al. (2015), is able to return a position in that sentence that corresponds to a word related to the word currently on top of the stack. They take advantage of this and propose a novel transition system that follows a top-down depth-first strategy to perform the syntactic analysis. Concretely, it considers the word pointed by the neural network as the child of the word on top of the stack, and builds the corresponding dependency relation between them. This results in a transition-based algorithm that can process unrestricted non-projective sentences in $O(n^{2})$ time complexity and requires 2$n$-1 actions to successfully parse a sentence with $n$ words.

We also take advantage of pointer network capabilities and use the neural network architecture introduced by Ma et al. (2018) to design a non-projective left-to-right transition-based algorithm, where the position value pointed by the network has the opposite meaning: it denotes the index that corresponds to the head node of the current focus word. This results in a straightforward transition system that can parse a sentence in just $n$ actions, without the need of any additional data structure and by just attaching each word from the sentence to another word (including the root node). Apart from increasing the parsing speed twofold (while keeping the same quadratic time complexity), it achieves the best accuracy to date among fully-supervised single-model dependency parsers on the PTB-SD, and obtains competitive accuracies on twelve different languages in comparison to the original top-down version.

2 Preliminaries

Ma et al. (2018) propose a novel neural network architecture whose main backbone is a pointer network Vinyals et al. (2015). This kind of neural networks are able to learn the conditional probability of a sequence of discrete numbers that correspond to positions in an input sequence (in this case, indexes of words in a sentence) and, by means of attention Bahdanau et al. (2014); Luong et al. (2015), implement a pointer that selects a position from the input at decoding time.

Their approach initially reads the whole sentence, composed of the $n$ words $w_{1},\dots,w_{n}$, and encodes each $w_{i}$ one by one into an encoder hidden state $e_{i}$. As encoder, they employ a combination of CNNs and bi-directional LSTMs Chiu and Nichols (2016); Ma and Hovy (2016). For each word, CNNs are used to obtain its character-level representation that is concatenated to the word and PoS embeddings to finally be fed into BiLSTMs that encode word context information.

As decoder they present a top-down transition system, where parsing configurations use the classic data structures Nivre (2008): a buffer (that contains unattached words) and a stack (that holds partially processed words).

The available parser actions are two transitions that we call $\mathsf{Shift\mathchar 45\relax Attach\mathchar 45\relax\textit{p}}$ and $\mathsf{Reduce}$. Given a configuration with word $w_{i}$ on top of the stack, as the pointer network just returns a position $p$ from a given sentence, they proceed as follows to determine which transition should be applied:

• •

If $p\neq i$, then the pointed word $w_{p}$ is considered as a child of $w_{i}$; so the parser chooses a $\mathsf{Shift\mathchar 45\relax Attach\mathchar 45\relax\textit{p}}$ transition to move $w_{p}$ from the buffer to the stack and build an arc $w_{i}\rightarrow w_{p}$.

• •

On the other hand, if $p=i$, then $w_{i}$ is considered to have found all its children, and a $\mathsf{Reduce}$ transition is applied to pop the stack.

The parsing process starts with a dummy root $on the stack and, by applying 2$n$-1 transitions, a dependency tree is built for the input in a top-down depth-first fashion, where multiple children of a same word are forced during training to be created in an inside-out manner. More in detail, for each parsing configuration$c_{t}$, the decoder (implemented as a uni-directional LSTM) receives the encoder hidden state$e_{i}$of the word$w_{i}$on top of the stack to generate a *decoder hidden state*$d_{t}$. After that,$d_{t}$, together with the sequence$s_{i}$of encoder hidden states from words still in the buffer plus$e_{i}$, are used to compute the attention vector$a^{t}$ as follows:

[TABLE]

As attention scoring function ($score()$), they adopt the biaffine attention mechanism described in Luong et al. (2015); Dozat and Manning (2016). Finally, the attention vector $a^{t}$ will be used to return the highest-scoring position $p$ and choose the next transition. The parsing process ends when only the root remains on the stack.

As extra high-order features, Ma et al. (2018) add grandparent and sibling information, whose encoder hidden states are added to that of the word on top of the stack to generate the corresponding decoder hidden state $d_{t}$. They prove that these additions improve final accuracy, especially when children are attached in an inside-out fashion.

According to the authors, the original stack-pointer network is trained to maximize the likelihood of choosing the correct word for each possible top-down path from the root to a leaf. More in detail, a dependency tree can be represented as a sequence of top-down paths $p_{1},\dots,p_{k}$, where each path $p_{i}$ corresponds to a sequence of words \,w_{i,1},w_{i,2},\dots,w_{i,l_{i}}$from the root to a leaf. Thus, the conditional probability$P_{\theta}(y|x)$of the dependency tree$y$for an input sentence$x$ can be factorized according to this top-down structure as:

[TABLE]

where $\theta$ represents model parameters, $p_{<i}$ stands for previous paths already explored, $w_{i,j}$ denotes the $j$th word in path $p_{i}$ and $w_{i,<j}$ represents all the previous words on $p_{i}$.

For more thorough details of the stack-pointer network architecture and the top-down transition system, please read the original work by Ma et al. (2018).

3 Our approach

We take advantage of the neural network architecture designed by Ma et al. (2018) and introduce a simpler left-to-right transition system that requires neither a stack nor a buffer to process the input sentence and where, instead of selecting a child of the word on top of the stack, the network points to the parent of the current focus word.

In particular, in our proposed approach, the parsing configuration just corresponds to a focus word pointer $i$, that is used to point to the word currently being processed. The decoding process starts with $i$ pointing at the first word of the sentence and, at each parsing configuration, only one action is available: the parameterized $\mathsf{Attach\mathchar 45\relax\textit{p}}$ transition, that links the focus word $w_{i}$ to the head word $w_{p}$ in position $p$ of the sentence (producing the dependency arc $w_{p}\rightarrow w_{i}$) and moves $i$ one position to the right. Note that, in our algorithm, $p$ can equal 0, attaching, in that case, $w_{i}$ to the dummy root node. The parsing process ends when the last word from the sentence is attached. This can be easily represented as a loop that traverses the input sentence from left to right, linking each word to another from the same sentence or to the dummy root. Therefore, we just need $n$ steps to process the $n$ words of a given sentence and build a dependency tree.

While our novel transition system intrinsically holds the single-head constraint (since, after attaching the word $w_{i}$, $i$ points to the next word $w_{i+1}$ in the sentence), it can produce an output with cycles.111In practice, even with the cycle detection mechanism disabled, the presence of cycles in output parses is very uncommon (for instance, just in 1% of sentences in the PTB-SD dev set) since our system seems to adequately model well-formed tree structures. Therefore, in order to build a well-formed dependency tree during decoding, attachments that generate cycles in the already-built dependency graph must be forbidden. Please note that the need of a cycle-checking extension does not increase the overall quadratic runtime complexity of the original implementation by Ma et al. (2018) since, as in other transition-based parsers such as Covington (2001); Gómez-Rodríguez and Nivre (2010), cycles can be incrementally identified in amortized constant time by keeping track of connected components using path compression and union by rank. Therefore, the left-to-right algorithm requires $n$ steps to produce a parse. In addition, at each step, the attention vector $a_{t}$ needs to be computed and cycles must be checked, both in $O(n)+O(n)=O(n)$ runtime. This results in a $O(n^{2})$ time complexity for decoding.222A practically faster version of the left-to-right parser might be implemented by just ignoring the presence of cycles during decoding, and destroying the cycles generated as a post-processing step that simply removes one of the arcs involved.

On the other hand, while in the top-down decoding only available words in the buffer (plus the word on top of the stack) can be pointed to by the network and they are reduced as arcs are created (basically to keep the single-head constraint); our proposed approach is less rigid: all words from the sentence (including the root node and excluding $w_{i}$) can be pointed to, as long as they satisfy the acyclicity constraint. This is necessary because two different words might be attached to the same head node and the latter can be located in the sentence either before or after $w_{i}$. Therefore, the sequence $s_{i}$, required by the attention score function (Eq.(1)), is composed of the encoder hidden states of all words from the input, excluding $e_{i}$, and prepending a special vector representation denoting the root node.

We also add extra features to represent the current focus word. Instead of using grandparent and sibling information (more beneficial for a top-down approach), we just add the encoder hidden states of the previous and next words in the sentence to generate $d_{t}$, which seems to be more suitable for a left-to-right decoding.

In dependency parsing, a tree for an input sentence of length $n$ can be represented as a set of $n$ directed and binary links $l_{1},\dots,l_{n}$. Each link $l_{i}$ is characterized by the word $w_{i}$ in position $i$ in the sentence and its head word $w_{h}$, resulting in a pair $(w_{i},w_{h})$. Therefore, to train this novel variant, we factorize the conditional probability $P_{\theta}(y|x)$ to a set of head-dependent pairs as follows:

[TABLE]

Therefore, the left-to-right parser is trained by maximizing the likelihood of choosing the correct head word $w_{h}$ for the word $w_{i}$ in position $i$, given the previous predicted links $l_{<i}$.

Finally, following a widely-used approach (also implemented in Ma et al. (2018)), dependency labels are predicted by a multiclass classifier, which is trained in parallel with the parser by optimizing the sum of their objectives.

4 Experiments

4.1 Data and Settings

We use the same implementation as Ma et al. (2018) and conduct experiments on the Stanford Dependencies de Marneffe and Manning (2008) conversion (using the Stanford parser v3.3.0)333https://nlp.stanford.edu/software/lex-parser.shtml of the English Penn Treebank Marcus et al. (1993), with standard splits and predicted PoS tags. In addition, we compare our approach to the original top-down parser on the same twelve languages from the Universal Dependency Treebanks444http://universaldependencies.org (UD) that were used by Ma et al. (2018).555Please note that, since they used a former version of UD datasets, we reran also the top-down algorithm on the latest treebank version (2.2) in order to perform a fair comparison.

Following standard practice, we just exclude punctuation for evaluating on PTB-SD and, for each experiment, we report the average Labelled and Unlabelled Attachment Scores (LAS and UAS) over 3 and 5 repetitions for UD and PTB-SD, respectively.

Finally, we use the same hyper-parameter values, pre-trained word embeddings and beam size (10 for PTB-SD and 5 for UD) as Ma et al. (2018).

4.2 Results

By outperforming the two current state-of-the-art graph-based Dozat and Manning (2016) and transition-based Ma et al. (2018) models on the PTB-SD, our approach becomes the most accurate fully-supervised dependency parser developed so far, as shown in Table 1.666It is worth mentioning that all parsers reported in this section make use of pre-trained word embeddings previously learnt from corpora beyond the training dataset. However, it is common practice in the literature that systems that only use standard pre-trained word embeddings are classed as fully-supervised models, even though, strictly, they are not trained exclusively on the official training data.

In addition, in Table 2 we can see how, under the exactly same conditions, the left-to-right algorithm improves over the original top-down variant in nine out of twelve languages in terms of LAS, obtaining competitive results in the remaining three datasets.

Finally, in spite of requiring a cycle-checking procedure, our approach proves to be twice as fast as the top-down alternative in decoding time, achieving, under the exact same conditions, a 23.08-sentences-per-second speed on the PTB-SD compared to 10.24 of the original system.777Please note that the implementation by Ma et al. (2018), also used by our novel approach, was not optimized for speed and, therefore, the reported speeds are just intended for comparing algorithms implemented under the same framework, but not to be considered as the best speed that a pointer-network-based system can potentially achieve.

5 Related work

There is previous work that proposes to implement dependency parsing by independently selecting the head of each word in a sentence, using neural networks. In particular, Zhang et al. (2017) make use of a BiLSTM-based neural architecture to compute the probability of attaching each word to one of the other input words, in a similar way as pointer networks do. During decoding, a post-processing step is needed to produce well-formed trees by means of a maximum spanning tree algorithm. Our approach does not need this post-processing, as cycles are forbidden during parsing instead, and achieves a higher accuracy thanks to the pointer network architecture and the use of information about previous dependencies.

Before Ma et al. (2018) presented their top-down parser, Chorowski et al. (2017) had already employed pointer networks Vinyals et al. (2015) for dependency parsing. Concretely, they developed a pointer-network-based neural architecture with multitask learning able to perform pre-processing, tagging and dependency parsing exclusively by reading tokens from an input sentence, without needing POS tags or pre-trained word embeddings. Like our approach, they also use the capabilities provided by pointer networks to undertake the parsing task as a simple process of attaching each word as dependent of another. They also try to improve the network performance with POS tag prediction as auxiliary task and with different approaches to perform label prediction. They do not exclude cycles, neither by forbidding them at parsing time or by removing them by post-processing, as they report that their system produces parses with a negligible amount of cycles, even with greedy decoding (matching our observation for our own system, in our case with beam-search decoding). Finally, the system developed by Chorowski et al. (2017) is constrained to projective dependencies, while our approach can handle unrestricted non-projective structures.

6 Conclusion

We present a novel left-to-right dependency parser based on pointer networks. We follow the same neural network architecture as the stack-pointer-based approach developed by Ma et al. (2018), but just using a focus word index instead of a buffer and a stack. Apart from doubling their system’s speed, our approach proves to be a competitive alternative on a variety of languages and achieves the best accuracy to date on the PTB-SD.

The good performance of our algorithm can be explained by the shortening of the transition sequence length. In fact, it has been proved by several studies Fernández-González and Gómez-Rodríguez (2012); Qi and Manning (2017); Fernández-González and Gómez-Rodríguez (2018) that by reducing the number of applied transitions, the impact of error propagation is alleviated, yielding more accurate parsers.

Our system’s source code is freely available at https://github.com/danifg/Left2Right-Pointer-Parser.

Acknowledgments

This work has received funding from the European Research Council (ERC), under the European Union’s Horizon 2020 research and innovation programme (FASTPARSE, grant agreement No 714150), from MINECO (FFI2014-51978-C2-2-R, TIN2017-85160-C2-1-R) and from Xunta de Galicia (ED431B 2017/01).