Contextualization of Morphological Inflection

TL;DR

This paper introduces a neural hybrid model for predicting fully inflected sentences from lemmatized input by inferring morphological features from context, improving natural language generation across diverse languages.

Contribution

It presents a novel hybrid model that explicitly reconstructs morphological features from context before inflection, advancing morphological prediction methods.

Findings

Hybrid model outperforms direct prediction in multiple languages.

Incorporating linguistic features improves inflection accuracy.

Model demonstrates robustness across typologically diverse languages.

Abstract

Critical to natural language generation is the production of correctly inflected text. In this paper, we isolate the task of predicting a fully inflected sentence from its partially lemmatized version. Unlike traditional morphological inflection or surface realization, our task input does not provide ``gold'' tags that specify what morphological features to realize on each lemmatized word; rather, such features must be inferred from sentential context. We develop a neural hybrid graphical model that explicitly reconstructs morphological features before predicting the inflected forms, and compare this to a system that directly predicts the inflected forms without relying on any morphological annotation. We experiment on several typologically diverse languages from the Universal Dependencies treebanks, showing the utility of incorporating linguistically-motivated latent variables into NLP models.

Tables

Table 1. Table 1: Example data entry: the target word be should be properly inflected into were to fit the sentential context.

Context:	two	cats	---	sitting
Lemmata:	two	cat	be	sit
Tags:	POS=NUM	POS=NOUN Num=Plur	---	POS=VERB Tense=Pres VerbForm=Part
Target:	two	cats	were	sitting

Equations

$$p(\mathbf{w},\mathbf{m}\mid \mathbf{\ell })=\left(\prod _{i=1}^{n}p(w_i\mid {\ell }_i,m_i)\right)p(\mathbf{m}\mid \mathbf{\ell })$$

$$p(\mathbf{m}\mid \mathbf{\ell })=\frac{1}{Z(\mathbf{\ell })}\prod _{i=1}^n\psi \left(m_i,m_{i-1},\mathbf{\ell }\right)$$

$$p(c_j\mid {\mathbf{c}}_{<j},$$

$$\text{softmax}\left(\mathbf{W}\cdot \varphi ({\mathbf{z}}_1,\dots ,{\mathbf{z}}_j)+\mathbf{b}\right)$$

$$p(\mathbf{c}\mid {\ell }_i,m_i)=\prod _{j=1}^{|w_i|}p(c_j\mid {\mathbf{c}}_{<j},{\ell }_i,m_i)$$

Full text

1 Introduction

author=ryan,color=violet!40,size=,fancyline,caption=,]Adjust intro to only discuss the task presented in the shared task. NLP systems are often required to generate grammatical text, e.g., in machine translation, summarization, dialogue, and grammar correction. One component of grammaticality is the use of contextually appropriate closed-class morphemes. In this work, we study contextual inflection, which has been recently introduced in the CoNLL-SIGMORPHON 2018 shared task Cotterell et al. (2018) to directly investigate context-dependent morphology in NLP. There, a system must inflect partially lemmatized tokens in sentential context. For example, in English, the system must reconstruct the correct word sequence two cats are sitting from partially lemmatized sequence two cat are sitting.author=trevor,color=green!40,size=,fancyline,caption=,]The partially lemmatised example is misleading, as we fully lemmatise. Among other things, this requires: (1) identifying cat as a noun in this context, (2) recognizing that cat should be inflected as plural to agree with the nearby verb and numeral, and (3) realizing this inflection as the suffix s. Most past work in supervised computational morphology—including the previous CoNLL-SIGMORPHON shared tasks on morphological reinflection Cotterell et al. (2017)—has focused mainly on step (3) above.

As the task has been introduced into the literature only recently, we provide some background. Contextual inflection amounts to a highly constrained version of language modeling. Language modeling predicts all words of a sentence from scratch, so the usual training and evaluation metric—perplexity—is dominated by the language model’s ability to predict content, which is where most of the uncertainty lies. Our task focuses on just the ability to reconstruct certain missing parts of the sentence—inflectional morphemes and their orthographic realization. This refocuses the modeling effort from semantic coherence to morphosyntactic coherence, an aspect of language that may take a back seat in current language models (see Linzen et al., 2016; Belinkov et al., 2017). Contextual inflection does not perfectly separate grammaticality modeling from content modeling: as illustrated in  Tab. 1, mapping two cats be sitting author=trevor,color=green!40,size=,fancyline,caption=,]Again partial. Why not two cat be sit? to the fully-inflected two cats were sitting does not require full knowledge of English grammar—the system does not have to predict the required word order nor the required auxiliary verb be, as these are supplied in the input. Conversely, this example does still require predicting some content—the semantic choice of past tense is not given by the input and must be guessed by the system.111This morphological feature is inherent in the sense of Booij (1996).

The primary contribution of this paper is a novel structured neural model for contextual inflection. The model first predicts the sequence of morphological tags from the partially lemmatized sequence and, then, it uses the predicted tag and lemma to inflect the word. We use this model to evince a simple point: models are better off jointly predicting morphological tags from context than directly learning to inflect lemmata from sentential context. Indeed, none of the participants in the 2018 shared task jointly predicted tags with the inflected forms. Comparing our new model to several competing systems, we show our model has the best performance on the majority of languages. We take this as evidence that predicting morphological tags jointly with inflecting is a better method for this task. Furthermore, we provide an analysis discussing the role of morphological complexity in model performance.

2 Joint Tagging and Inflection

Given a language, let $\cal M$ be a set of morphological tags in accordance with the Universal Dependencies annotation Nivre et al. (2016). Each $m\in\cal M$ has the form $m=\langle t,\sigma\rangle$, where $t$ is a part of speech, and the slot $\sigma$ is a set of attribute–value pairs that represent morphosyntactic information, such as number, case, tense, gender, person, and others. We take $t\in{\cal T}$, the set of universal parts of speech described by Petrov et al. (2012). A sentence consists of a finite word sequence $\mathbf{w}$ (we use boldface for sequence variables). For every word $w_{i}$ in the sequence, there is a corresponding analysis in terms of a morphological tag $m_{i}\in\cal M$ and a lemma $\ell_{i}$. In general, $w_{i}$ is determined by the pair $\langle\ell_{i},m_{i}\rangle$.222Although $w_{i}$ can sometimes be computed by concatenating $\ell_{i}$ with $m_{i}$-specific affixes, it can also be irregular. Using this notation, Cotterell et al. (2018)’s shared task is to predict a sentence $\mathbf{w}$ from its partially lemmatized form $\boldsymbol{\ell}$, inferring $\mathbf{m}$ as an intermediate latent variable. Our dataset (§ 3) has all three sequences for each sentence.

2.1 A Structured Neural Model

author=ryan,color=violet!40,size=,fancyline,caption=,]Is it fair to call it a joint model? Discuss reviewer comments. Consider an extreme case when all words are lemmatized.333In case of partially lemmatized sequence we still train the model to predict the tags over the entire sequence, but evaluate it only for lemmatized slots. We introduce a structured neural model for contextual inflection, as follows:

[TABLE]

In other words, the distribution is over interleaved sequences of one-to-one aligned inflected words and morphological tags, conditioned on a lemmatized sequence, all of length $n$. This distribution is drawn as a hybrid (directed–undirected) graphical model Koller and Friedman (2009) in Fig. 1. We define the two conditional distributions in the model in § 2.2 and § 2.3, respectively.

2.2 A Neural Conditional Random Field

The distribution $p(\mathbf{m}\mid\boldsymbol{\ell})$ is defined to be a conditional random field (CRF; Lafferty et al., 2001). In this work, our CRF is a conditional distribution over morphological taggings of an input sequence. We define this conditional distribution as

[TABLE]

where $\psi(\cdot,\cdot,\cdot)\geq 0$ is an arbitrary potential, $Z(\boldsymbol{\ell})$ normalizes the distribution, and $m_{0}$ is a distinguished start-of-sequence symbol.

In this work, we opt for a recurrent neural potential—specifically, we adopt a parameterization similar to the one given by Lample et al. (2016). Our potential $\psi$ is computed as follows. First, the sequence $\boldsymbol{\ell}$ is encoded into a sequence of word vectors using the strategy described by Ling et al. (2015): word vectors are passed to a bidirectional LSTM Graves et al. (2005), where the corresponding hidden states are concatenated at each time step. We simply refer to the hidden state $\mathbf{h}_{i}\in\mathbb{R}^{d}$ as the result of said concatenation at the $i$-th step. Using $\mathbf{h}_{i}$, we can define the potential function as $\psi\left(m_{i},m_{i-1}\right)=\exp\left(A_{m_{i},m_{i-1}}+\mathbf{o}_{m_{i}}^{\top}\mathbf{h}_{i}\right),$ where $A_{m_{i},m_{i-1}}$ is a transition weight matrix and $\mathbf{o}_{m_{i}}\in\mathbb{R}^{d}$ is a morphological tag embedding; both are learned.

2.3 The Morphological Inflector

The conditional distribution $p(w_{i}\mid\ell_{i},m_{i})$ is parameterized by a neural encoder–decoder model with hard attention from Aharoni and Goldberg (2017). The model was one of the top performers in the 2016 SIGMORPHON shared task Cotterell et al. (2016); it achieved particularly high accuracy in the low-resource setting. Hard attention is motivated by the observation that alignment between the input and output sequences is often monotonic in inflection tasks. In the model, the input lemma is treated as a sequence of characters, and encoded using a bidirectional LSTM Graves and Schmidhuber (2005), to produce vectors $\mathbf{x}_{j}$ for each character position $j$. Next the word $w_{i}=\mathbf{c}=c_{1}\cdots c_{|w_{i}|}$ is generated in a decoder character-by-character:

[TABLE]

where $\mathbf{z}_{j}$ is the concatenation of the current attended input $\mathbf{x}_{j}$ alongside morphological features, $m_{i}$, and an embedding of the previously generated symbol $c_{j-1}$; and finally $\phi$ is an LSTM over the sequence of $\mathbf{z}_{j}$ vectors. The decoder additionally predicts a type of operation.444The model can be viewed as a transition system trained over aligned character-level strings to learn sequences of operations (write or step). author=jason,color=orange!40,size=,fancyline,caption=,]this description overall is super hard to understand and needs to be rewritten, but one confusion is that you are indexing here with $i$ (actually $\mathbf{i}$), which indexes words. Don’t you mean $j$, which indexes the chars within word $i$? The distribution in Eq. 3, strung together with the other conditionals, yields a joint distribution over the entire character sequence:

[TABLE]

author=ryan,color=violet!40,size=,fancyline,caption=,]Where did $\ell_{i}$ come from? author=trevor,color=green!40,size=,fancyline,caption=,]I added $\ell_{i}$ to conditioning on right, surely it’s needed

For instance, to map the lemma talk to its past form talked, we feed in POS=V;Tense=PAST <w> t a l k </w> and train the network to output <w> t a l k e d </w>, where we have augmented the orthographic character alphabet $\Sigma$ with the feature–attribute pairs that constitute the morphological tag $m_{i}$.

2.4 Parameter Estimation and Decoding

We optimize the log-likelihood of the training data with respect to all model parameters. As Eq. 1 is differentiable, this is achieved with standard gradient-based methods. For decoding we use a greedy strategy where we first decode the CRF, that is, we solve the problem $\mathbf{m}^{\star}=\text{argmax}_{\mathbf{m}}\log p(\mathbf{m}\mid\boldsymbol{\ell})$, using the Viterbi (1967) algorithm. We then use this decoded $\mathbf{m}^{\star}$ to generate forms from the inflector. Note that finding the one-best string under our neural inflector is intractable, and for this reason we use greedy search.

3 Experiments

Dataset.

We use the Universal Dependencies v1.2 dataset Nivre et al. (2016) for our experiments. We include all the languages with information on their lemmata and fine-grained grammar tag annotation that also have fasttext embeddings Bojanowski et al. (2017), which are used for word embedding initialization.555We also choose mainly non-Wikipedia datasets to reduce any possible intersection with the data used for the FastText model training

Evaluation.

We evaluate our model’s ability to predict: (i) the correct morphological tags from the lemma context, and (ii) the correct inflected forms. As our evaluation metric, we report 1-best accuracy for both tags and word form prediction.

Configuration.

We use a word and character embedding dimensionality of 300 and 100, respectively. The hidden state dimensionality is set to 200. All models are trained with Adam Kingma and Ba (2014), with a learning rate of 0.001 for 20 epochs.

Baselines.

We use two baseline systems: (1) the CoNLL–SIGMORPHON 2018 subtask 2 neural encoder–decoder with an attention mechanism (“SM”; Cotterell et al. (2018)), where the encoder represents a target form context as a concatenation of its lemma, its left and right word forms, their lemmata and tag representations, and then the decoder generates the target inflected form character-by-character; and (2) a monolingual version of the best performing system of the shared task (“CPH”; Kementchedjhieva et al. (2018)) that augments the above encoder–decoder with full (sentence-level) left and right contexts (comprising of forms, their lemmata and morphological tags) as well as predicts morphological tags for a target form as an auxiliary task.666It has been shown to improve the model’s performance. In both cases, the hyperparameters are set as described in Cotterell et al. (2018). We additionally evaluate the SIGMORPHON baseline system on prediction of the target form without any information on morphological tags (“direct”).

4 Results and Discussion

Tab. 2 presents the accuracy of our best model across all languages.777The accuracy numbers are on average higher than the ones achieved in terms of the CoNLL–SIGMORPHON 2018 subtask 2 since we did not filter out tokens that are typically not inflected (such as articles or prepositions). Below we highlight two main lessons from our error analysis that apply to a wider range of generation tasks, e.g., machine translation and dialog systems.

author=kat,color=green!40,size=,fancyline,caption=,]Tab2: ADD discussion: results on Italian and Hindi are interesting; both languages have high morphological tagging accuracies, but low performance on the final accuracy, even compared to the DIRECT model.

Directly Predicting Morphology.

Tab. 2 indicates that all systems that make use of morphological tags outperform the direct baseline on most languages. The comparison of our hybrid model with latent morphological tags to the direct form generation baseline in SM suggests that we should be including linguistically-motivated latent variables into models of natural language generation. We observe in Tab. 2 that predicting the tag together with the form (joint) often improves performance. The most interesting comparison here is with the multi-task CPH method, which includes morphology into the model without joint modeling; our model achieves higher results on 7/10 languages.

Morphological Complexity Matters.

We observed that for languages with rich case systems, e.g., the Slavic languages (which exhibit a lot of fusion), the agglutinative Finno-Ugric languages, and Basque, performance is much worse. These languages present a broader decision space and often require inferring which morphological categories need to be in agreement in order to make an accurate prediction. This suggests that generation in languages with more morphological complexity will be a harder problem for neural models to solve. Indeed, this problem is under-explored, as the field of NLP tends to fixate on generating English text, e.g., in machine translation or dialogue system research.

Error Analysis.

We focused error analysis on prediction of agreement categories. Our analysis of adjective–noun agreement category prediction suggests that our model is able to infer adjective number, gender, and case from its head noun. Verb gender, which appears only in the past tense of many Slavic languages, seems to be harder to predict. Given that the linear distance between the subject and the verb may be longer, we suspect the network struggles to learn longer-distance dependencies, consistent with the findings of Linzen et al. (2016). Overall, automatic inference of agreement categories is an interesting problem that deserves more attention, and we leave it for future work.

We also observe that most uncertainty comes from morphological categories such as noun number, noun definiteness (which is expressed morphologically in Bulgarian), and verb tense, all of which are inherent Booij (1996)888Such categories exist in most languages that exhibit some degree of morphological complexity. and typically cannot be predicted from sentential context if they do not participate in agreement.999Unless there is no strong signal within a sentence such as yesterday, tomorrow, or ago as in the case of tense. On the other hand, aspect, although being closely related to tense, is well-predicted since it is mainly expressed as a separate lexeme. But, in general, it is still problematic to make a prediction in languages where aspect is morphologically marked or highly mixed with tense as in Basque.

We additionally compared 1-best and 10-best predictions for tags. Most mispredictions existing in 1-best lists are due to inherent categories mentioned above (that allow multiple plausible options that can fit the sentential context). Indeed, the problem is usually solved by allowing system to output 10-best lists. There, precision@10 is on average 8 points higher than precision@1.

Finally, our analysis of case category prediction on nouns shows that more common cases such as the nominative, accusative, and genitive are predicted better, especially in languages with fixed word order. On the other hand, cases that appear less frequently and on shifting positions (such as the instrumental), as well as those not associated with specific prepositions, are less well predicted. In addition, we evaluated the model’s performance when all forms are replaced by their corresponding lemmata (as in two cat be sit). For freer word order languages such as Polish or Latin, we observe a substantial drop in performance because most information on inter-word relations and their roles (expressed by means of case system) is lost.

5 Related Work

The primary evaluation for most contemporary language and translation modeling research is perplexity, BLEU Papineni et al. (2002), or METEOR Banerjee and Lavie (2005). Undoubtedly, such metrics are necessary for extrinsic evaluation and comparison. However, relatively few studies have focused on intrinsic evaluation of the model’s mastery of grammaticality. Recently, Linzen et al. (2016) investigated the ability of an LSTM language model to capture sentential structure, by evaluating subject–verb agreement with respect to number, and showed that under strong supervision, the LSTM is able to approximate dependencies.

Taking it from the other perspective, a truer measure of grammatical competence would be a task of mapping a meaning representation to text, where the meaning representation specifies all necessary semantic content—content lemmata, dependency relations, and “inherent” closed-class morphemes (semantic features such as noun number, noun definiteness, and verb tense)—and the system is to realize this content according to the morphosyntactic conventions of a language, which means choosing word order, agreement morphemes, function words, and the surface forms of all words. Such tasks have been investigated to some extent—generating text from tectogrammatical trees Hajic et al. (2002); Ptáček and Žabokrtský (2006) or from an AMR graph Song et al. (2017). Belz et al. (2011) organized a related surface realization shared task on mapping unordered and uninflected dependency trees to properly ordered inflected sentences. The generated sentences were afterwards assessed by human annotators, making the task less scalable and more time consuming. Although our task is not perfectly matched to grammaticality modeling, the upside is that it is a “lightweight” task that works directly on text. No meaning representation is required. Thus, training and test data in any language can be prepared simply by lemmatizing a naturally occurring corpus.

Finally, as a morphological inflection task, the form generation task is closely related to previous SIGMORPHON shared tasks Cotterell et al. (2016, 2017). There, most neural models achieve high accuracy on many languages at type-level prediction of the form from its lemma and slot. The current task is more challenging in that the model has to perform token-level form generation and inherently infer the slot from the contextual environment. Our findings are in line with those from the CoNLL-SIGMORPHON 2018 shared task Cotterell et al. (2018) and provide extra evidence of the utility of morphosyntactic features.

6 Conclusion

This work proposed a method for contextual inflection using a hybrid architecture. Evaluation over several diverse languages showed consistent improvements over state of the art. Our analysis demonstrated that the contextual inflection can be a highly challenging task, and the inclusion of morphological features prediction is an important element in such a system. We also highlighted two types of morphological categories, contextual and inherent, in which the former relies on agreement and the latter comes from a speaker’s intention.

Acknowledgements

We thank all anonymous reviewers for their comments. The first author would like to acknowledge the Google PhD fellowship. The second author would like to acknowledge a Facebook Fellowship.