Cross-lingual Abstract Meaning Representation Parsing

TL;DR

This paper introduces a cross-lingual AMR parsing method using annotation projection from English to multiple languages, and proposes an evaluation technique that does not require gold annotations in target languages.

Contribution

It presents a novel annotation projection approach for cross-lingual AMR parsing and a new evaluation method that leverages English gold annotations without needing target language gold standards.

Findings

Promising results for Italian, Spanish, German, and Chinese

Effective evaluation method without target language gold annotations

Successful cross-lingual transfer of AMR parsers

Abstract

Abstract Meaning Representation (AMR) annotation efforts have mostly focused on English. In order to train parsers on other languages, we propose a method based on annotation projection, which involves exploiting annotations in a source language and a parallel corpus of the source language and a target language. Using English as the source language, we show promising results for Italian, Spanish, German and Chinese as target languages. Besides evaluating the target parsers on non-gold datasets, we further propose an evaluation method that exploits the English gold annotations and does not require access to gold annotations for the target languages. This is achieved by inverting the projection process: a new English parser is learned from the target language parser and evaluated on the existing English gold standard.

Tables

Table 1. Table 1: Silver, gold and full-cycle Smatch scores for projection-based and MT-based systems.

	System	Silver	Gold	Cycle
IT	Projection	45	43	45
	Moses	51	52	51
	Nematus	49	43	41
	GT	52	58	59
ES	Projection	44	42	44
	Moses	53	53	51
	Nematus	51	43	42
	GT	56	60	60
DE	Projection	45	39	43
	Moses	50	49	49
	Nematus	47	38	39
	GT	54	57	59
ZH	Projection	45	35	32
	Moses	57	42	48
	Nematus	57	39	40
	GT	64	50	55

Table 2. Table 2: BLEU scores for Moses, Nematus and Google Translate (GT) on the (out-of-domain) LDC2015E86 test set

Model	Moses	Nematus	GT
EN-IT	23.83	21.27	61.31
IT-EN	23.74	19.77	42.20
EN-ES	29.00	26.14	78.14
ES-EN	27.66	21.63	50.78
EN-DE	15.47	15.74	63.48
DE-EN	21.50	14.96	41.78
EN-ZH	9.19	8.67	26.75
ZH-EN	10.81	10.37	22.21

Equations

$$\forall i,j,v{t}_j\in W(s_i)\wedge v\in A_s(s_i)\Rightarrow v\in A_t(t_j)$$

Full text

1 Introduction

Abstract Meaning Representation (AMR) parsing is the process of converting natural language sentences into their corresponding AMR representations Banarescu et al. (2013). An AMR is a graph with nodes representing the concepts of the sentence and edges representing the semantic relations between them. Most available AMR datasets large enough to train statistical models consist of pairs of English sentences and AMR graphs.

The cross-lingual properties of AMR across languages has been the subject of preliminary discussions. The AMR guidelines state that AMR is not an interlingua Banarescu et al. (2013) and Bojar (2014) categorizes different kinds of divergences in the annotation between English AMRs and Czech AMRs. Xue et al. (2014) show that structurally aligning English AMRs with Czech and Chinese AMRs is not always possible but that refined annotation guidelines suffice to resolve some of these cases. We extend this line of research by exploring whether divergences among languages can be overcome, i.e., we investigate whether it is possible to maintain the AMR annotated for English as a semantic representation for sentences written in other languages, as in Figure 1.

We implement AMR parsers for Italian, Spanish, German and Chinese using annotation projection, where existing annotations are projected from a source language (English) to a target language through a parallel corpus (e.g., Yarowsky et al., 2001; Hwa et al., 2005; Padó and Lapata, 2009; Evang and Bos, 2016). By evaluating the parsers and manually analyzing their output, we show that the parsers are able to recover the AMR structures even when there exist structural differences between the languages, i.e., although AMR is not an interlingua it can act as one. This method also provides a quick way to prototype multilingual AMR parsers, assuming that Part-of-speech (POS) taggers, Named Entity Recognition (NER) taggers and dependency parsers are available for the target languages. We also propose an alternative approach, where Machine Translation (MT) is used to translate the input sentences into English so that an available English AMR parser can be employed. This method is an even quicker solution which only requires translation models between the target languages and English.

Due to the lack of gold standard in the target languages, we exploit the English data to evaluate the parsers for the target languages. (Henceforth, we will use the term target parser to indicate a parser for a target language.) We achieve this by first learning the target parser from the gold standard English parser, and then inverting this process to learn a new English parser from the target parser. We then evaluate the resulting English parser against the gold standard. We call this “full-cycle” evaluation.

Similarly to Evang and Bos (2016), we also directly evaluate the target parser on “silver” data, obtained by parsing the English side of a parallel corpus.

In order to assess the reliability of these evaluation methods, we collected gold standard datasets for Italian, Spanish, German and Chinese by acquiring professional translations of the AMR gold standard data to these languages. We hypothesize that the full-cycle score can be used as a more reliable proxy than the silver score for evaluating the target parser. We provide evidence to this claim by comparing the three evaluation procedures (silver, full-cycle, and gold) across languages and parsers.

Our main contributions are:

• •

We provide evidence that AMR annotations can be successfully shared across languages.

• •

We propose two ways to rapidly implement non-English AMR parsers.

• •

We propose a novel method to evaluate non-English AMR parsers when gold annotations in the target languages are missing. This method highly correlates with gold standard evaluation, obtaining a Pearson correlation coefficient of 0.95.

• •

We release human translations of an AMR dataset (LDC2015E86) to Italian, Spanish, German and Chinese.

2 Cross-lingual AMR parsing

AMR is a semantic representation heavily biased towards English, where labels for nodes and edges are either English words or Propbank frames Kingsbury and Palmer (2002). The goal of AMR is to abstract away from the syntactic realization of the original sentences while maintaining its underlying meaning. As a consequence, different phrasings of one sentence are expected to provide identical AMR representations. This canonicalization does not always hold across languages: two sentences that express the same meaning in two different languages are not guaranteed to produce identical AMR structures Bojar (2014); Xue et al. (2014). However, Xue et al. (2014) show that in many cases the unlabeled AMRs are in fact shared across languages. We are encouraged by this finding and argue that it should be possible to develop algorithms that account for some of these differences when they arise. We therefore introduce a new problem, which we call cross-lingual AMR parsing: given a sentence in any language, the goal is to recover the AMR graph that was originally devised for its English translation. This task is harder than traditional AMR parsing as it requires to recover English labels as well as to deal with structural differences between languages, usually referred as translation divergence. We propose two initial solutions to this problem: by annotation projection and by machine translation.

2.1 Method 1: Annotation Projection

AMR is not grounded in the input sentence, therefore there is no need to change the AMR annotation when projecting to another language. We think of English labels for the graph nodes as ones from an independent language, which incidentally looks similar to English. However, in order to train state-of-the-art AMR parsers, we also need to project the alignments between AMR nodes and words in the sentence (henceforth called AMR alignments). We use word alignments, similarly to other annotation projection work, to project the AMR alignments to the target languages.

Our approach depends on an underlying assumption that we make: if a source word is word-aligned to a target word and it is AMR aligned with an AMR node, then the target word is also aligned to that AMR node. More formally, let $S=s_{1}\dots s_{|s|}$ be the source language sentence and $T=t_{1}\dots t_{|t|}$ be the target language sentence; $A_{s}(\cdot)$ be the AMR alignment mapping word tokens in $S$ to the set of AMR nodes that are triggered by it; $A_{t}(\cdot)$ be the same function for $T$; $v$ be a node in the AMR graph; and finally, $W(\cdot)$ be an alignment that maps a word in $S$ to a subset of words in $T$. Then, the AMR projection assumption is:

[TABLE]

In the example of Figure 1, Questa is word-aligned with This and therefore AMR-aligned with the node this, and the same logic applies to the other aligned words. The words is, the and of do not generate any AMR nodes, so we ignore their word alignments. We apply this method to project existing AMR annotations to other languages, which are then used to train the target parsers.

2.2 Method 2: Machine Translation

We invoke an MT system to translate the sentence into English so that we can use an available English parser to obtain its AMR graph. Naturally, the quality of the output graph depends on the quality of the translations. If the automatic translation is close to the reference translation, then the predicted AMR graph will be close to the reference AMR graph. It is therefore evident that this method is not informative in terms of the cross-lingual properties of AMR. However, its simplicity makes it a compelling engineering solution for parsing other languages.

2.3 Evaluation

We now turn to the problem of evaluation. Let us assume that we trained a parser for a target language, for example using the annotation projection method discussed in Section 2.1. In line with rapid development of new parsers, we assume that the only gold AMR dataset available is the one released for English.

SILVER

We can generate a silver test set by running an automatic (English) AMR parser on the English side of a parallel corpus and use the output AMRs as references. However, the silver test set is affected by mistakes made by the English AMR parser, therefore it may not be reliable.

FULL-CYCLE

In order to perform the evaluation on a gold test set, we propose full-cycle evaluation: after learning the target parser from the English parser, we invert this process to learn a new English parser from the target parser, in the same way that we learned the target parser from the English parser. The resulting English parser is then evaluated against the (English) AMR gold standard. We hypothesize that the score of the new English parser can be used as a proxy to the score of the target parser.

GOLD

To show whether the evaluation methods proposed can be used reliably, we also generated gold test AMR datasets for four target languages (Italian, Spanish, German and Chinese). In order to do so, we collected professional translations for the English sentences in the AMR test set.111These datasets are currently available upon request from the authors. We were then able to create pairs of human-produced sentences with human-produced AMR graphs.

A diagram summarizing the different evaluation stages is shown in Figure 2. In the case of MT-based systems, the full-cycle corresponds to first translating from English to the target language and then back to English (back-translation), and only then parsing the sentences with the English AMR parser. At the end of this process, a noisy version of the original sentence will be returned and its parsed graph will be a noisy version of the graph parsed from the original sentence.

3 Experiments

We run experiments on four languages: Italian, Spanish, German and Chinese. We use Europarl Koehn (2005) as the parallel corpus for Italian, Spanish and German, containing around 1.9M sentences for each language pair. For Chinese, we use the first 2M sentences from the United Nations Parallel Corpus Ziemski et al. (2016). For each target language we extract two parallel datasets of 20,000/2,000/2,000 (train/dev/test) sentences for the two step of the annotation projection (English $\rightarrow$ target and target $\rightarrow$ English). These are used to train the AMR parsers. The projection approach also requires training the word alignments, for which we use all the remaining sentences from the parallel corpora (Europarl for Spanish/German/Italian and UN Parallel Corpus for Chinese). These are also the sentences we use to train the MT models. The gold AMR dataset is LDC2015E86, containing 16,833 training sentences, 1,368 development sentences, and 1,371 testing sentences.

Word alignments were generated using fast_align Dyer et al. (2013), while AMR alignments were generated with JAMR Flanigan et al. (2014). AMREager Damonte et al. (2017) was chosen as the pre-existing English AMR parser. AMREager is an open-source AMR parser that needs only minor modifications for re-use with other languages. Our multilingual adaptation of AMREager is available at http://www.github.com/mdtux89/amr-eager-multilingual.222A demo is available at http://cohort.inf.ed.ac.uk/amreager.html. It requires tokenization, POS tagging, NER tagging and dependency parsing, which for English, German and Chinese are provided by CoreNLP Manning et al. (2014). We use Freeling Carreras et al. (2004) for Spanish, as CoreNLP does not provide dependency parsing for this language. Italian is not supported in CoreNLP: we use Tint Aprosio and Moretti (2016), a CoreNLP-compatible NLP pipeline for Italian.

In order to experiment with the approach of Section 2.2, we experimented with translations from Google Translate.333https://translate.google.com/toolkit. As Google Translate has access to a much larger training corpus, we also trained baseline MT models using Moses Koehn et al. (2007) and Nematus Sennrich et al. (2017), with the same training data we use for the projection method and default hyper-parameters.

Smatch Cai and Knight (2013) is used to evaluate AMR parsers. It looks for the best alignment between the predicted AMR and the reference AMR and it then computes precision, recall and $F_{1}$ of their edges. The original English parser achieves 65% Smatch score on the test split of LDC2015E86. Full-cycle and gold evaluations use the same dataset, while silver evaluation is performed on the split of the parallel corpora we reserved for testing. Results are shown in Table 1. The Google Translate system outperforms all other systems, but is not directly comparable to them, as it has the unfair advantage of being trained on a much larger dataset. Due to noisy JAMR alignments and silver training data involved in the annotation projection approach, the MT-based systems give in general better parsing results. The BLEU scores of all translation systems are shown in Table 2.

There are several sources of noise in the annotation projection method, which affect the parsing results: 1) the parsers are trained on silver data obtained by an automatic parser for English; 2) the projection uses noisy word alignments; 3) the AMR alignments on the source side are also noisy; 4) translation divergences exist between the languages, making it sometimes difficult to project the annotation without loss of information.

4 Qualitative Analysis

Figure 3 shows examples of output parses444In this section, all parsed graphs were generated with the projection-based system of Section 2.1. for all languages, including the AMR alignments by-product of the parsing process, that we use to discuss the mistakes made by the parsers.

In the Italian example, the only evident error is that Infine (Lastly) should be ignored. In the Spanish example, the word medida (measure) is wrongly ignored: it should be used to generate a child of the node impact-01. Some of the :ARG roles are also not correct. In the German example, meines (my) should reflect the fact that the speaker is talking about his own country. Finally, in the Chinese example, there are several mistakes including yet another concept identification mistake: intend-01 is erroneously triggered.

Most mistakes involve concept identification. In particular, relevant words are often erroneously ignored by the parser. This is directly related to the problem of noisy word alignments in annotation projection: the parser learns what words are likely to trigger a node (or a set of nodes) in the AMR by looking at their AMR alignments (which are induced by the word alignments). If an important word consistently remains unaligned, the parser will erroneously learn to discard it. More accurate alignments are therefore crucial in order to achieve better parsing results. We computed the percentage of words in the training data that are learned to be non-content-bearing in each parser and we found that the Chinese parser, which is our least accurate parser, is the one that most suffer from this, with 33% non-content-bearing words. On the other hand, in the German parser, which is the highest scoring, only 26% of the words are non-content-bearing, which is the lowest percentage amongst all parsers.

4.1 Translational Divergence

In order to investigate the hypothesis that AMR can be shared across these languages, we now look at translational divergence and discuss how it affects parsing, following the classification used in previous work Dorr et al. (2002); Dorr (1994), which identifies classes of divergences for several languages. Sulem et al. (2015) also follow the same categorization for French.

Figure 4 shows six sentences displaying these divergences. The aim of this analysis is to assess how the parsers deal with the different kind of translational divergences, regardless of the overall quality of the output.

Categorical.

This divergence happens when two languages use different POS tags to express the same meaning. For example, the English sentence I am jealous of you is translated into Spanish as Tengo envidia de ti (I have jealousy of you). The English adjective jealous is translated in the Spanish noun envidia. In Figure 4a we note that the categorical divergence does not create problems since the parsers correctly recognized that envidia (jealousy/envy) should be used as the predicate, regardless of its POS.

Conflational.

This divergence happens when verbs expressed in a language with a single word can be expressed with more words in another language. Two subtypes are distinguished: manner and light verb. Manner refers to a manner verb that is mapped to a motion verb plus a manner-bearing word. For example, We will answer is translated in the Italian sentence Noi daremo una riposta (We will give an answer), where to answer is translated as daremo una risposta (will give an answer). Figure 4b shows that the Italian parser generates a sensible output for this sentence by creating a single node labeled answer-01 for the expression dare una riposta.

In a light verb conflational divergence, a verb is mapped to a light verb plus an additional meaning unit, such as when I fear is translated as Io ho paura (I have fear) in Italian: to fear is mapped to the light verb ho (have) plus the noun paura (fear). Figure 4e shows that also this divergence is dealt properly by the Italian parser: ho paura correctly triggers the root fear-01.

Structural.

This divergence happens when verb arguments result in different syntactic configurations, for example, due to an additional PP attachment. When translating He entered the house with Lui è entrato nella casa (He entered in the house), the Italian translation has an additional in preposition. Also this parsed graph, in Figure 4c, is structurally correct. The missing node he is due to pronoun-dropping, which is frequent in Italian.

Head swapping.

This divergence occurs when the direction of the dependency between two words is inverted. For example, I like eating, where like is head of eating, becomes Ich esse gern (I eat likingly) in German, where the dependency is inverted. Unlike all other examples, in this case, the German parser does not cope well with this divergence: it is unable to recognize like-01 as the main concept in the sentence, as shown in Figure 4d.

Thematic.

Finally, the parse of Figure 4f has to deal with a thematic divergence, which happens when the semantic roles of a predicate are inverted. In the sentence I like grapes, translated to Spanish as Me gustan uvas, I is the subject in English while Me is the object in Spanish. Even though we note an erroneous reentrant edge between grape and I, the thematic divergence does not create problems: the parser correctly recognizes the :ARG0 relationship between like-01 and I and the :ARG1 relationship between like-01 and grape. In this case, the edge labels are important, as this type of divergence is concerned with the semantic roles.

5 Discussion

Can AMR be shared across these languages?

As mentioned in Section 2.2, the MT-based systems are not helpful in answering this question and we instead focus on the projection-based parsers. Qualitative analysis showed that the parsers are able to overcome translational divergence and that concept identification must be more accurate in order to provide good parsing results. We therefore argue that the suboptimal performance of the parsers in terms of Smatch scores is due to the many sources of noise in the annotation projection approach rather than instability of AMR across languages. We provide strong evidence that cross-lingual AMR parsing is indeed feasible and hope that the release of the gold standard test sets will motivate further work in this direction.

Are silver and full-cycle evaluations reliable?

We computed the Pearson correlation coefficients for the Smatch scores of Table 1 to determine how well silver and full-cycle correlate with gold evaluation. Full-cycle correlates better than silver: the Pearson coefficient is 0.95 for full-cycle and 0.47 for silver. Figure 5 shows linear regression lines. Unlike silver, full-cycle uses the same dataset as gold evaluation and it does not contain parsing mistakes, which makes it more reliable than silver. Interestingly, if we ignore the scores obtained for Chinese, the correlation between silver and gold dramatically increases, perhaps indicating that Europarl is more suitable than the UN corpus for this task: the Pearson coefficient becomes 0.97 for full-cycle and 0.87 for silver. A good proxy for gold evaluation should rank different systems similarly. We hence computed the Kendall-tau score Kendall (1945), a measure for similarity between permutations, of the rankings extracted from Table 1. The results further confirm that full-cycle approximate gold better than silver does: the score is 0.40 for silver and 0.82 for full-cycle. Full cycle introduces additional noise but it is not as expensive as gold and is more reliable than silver.

6 Related Work

AMR parsing for languages other than English has made only a few steps forward. In previous work Li et al. (2016); Xue et al. (2014); Bojar (2014), nodes of the target graph were labeled with either English words or with words in the target language. We instead use the AMR annotation used for English for the target language as well, without translating any word. To the best of our knowledge, the only previous work that attempts to automatically parse AMR graphs for non-English sentences is by Vanderwende et al. (2015). Sentences in several languages (French, German, Spanish and Japanese) are parsed into a logical representation, which is then converted to AMR using a small set of rules. A comparison with this work is difficult, as the authors do not report results for the parsers (due to the lack of an annotated corpus) or release their code.

Besides AMR, other semantic parsing frameworks for non-English languages have been investigated Hoffman (1992); Cinková et al. (2009); Gesmundo et al. (2009); Evang and Bos (2016). Evang and Bos (2016) is the most closely related to our work as it uses a projection mechanism similar to ours for CCG. A crucial difference is that, in order to project CCG parse trees to the target languages, they only make use of literal translation. Previous work has also focused on assessing the stability across languages of semantic frameworks such as AMR Xue et al. (2014); Bojar (2014), UCCA Sulem et al. (2015) and Propbank Van der Plas et al. (2010).

Cross-lingual techniques can cope with the lack of labeled data on languages when this data is available in at least one language, usually English. The annotation projection method, which we follow in this work, is one way to address this problem. It was introduced for POS tagging, base noun phrase bracketing, NER tagging, and inflectional morphological analysis Yarowsky et al. (2001) but it has also been used for dependency parsing Hwa et al. (2005), role labeling Padó and Lapata (2009); Akbik et al. (2015) and semantic parsing Evang and Bos (2016). Another common thread of cross-lingual work is model transfer, where parameters are shared across languages Zeman and Resnik (2008); Cohen and Smith (2009); Cohen et al. (2011); McDonald et al. (2011); Søgaard (2011).

7 Conclusions

We introduced the problem of parsing AMR structures, annotated for English, from sentences written in other languages as a way to test the cross-lingual properties of AMR. We provided evidence that AMR can be indeed shared across the languages tested and that it is possible to overcome translational divergences. We further proposed a novel way to evaluate the target parsers that does not require manual annotations of the target language. The full-cycle procedure is not limited to AMR parsing and could be used for other cross-lingual problems in NLP. The results of the projection-based AMR parsers indicate that there is a vast room for improvements, especially in terms of generating better alignments. We encourage further work in this direction by releasing professional translations of the AMR test set into four languages.

Acknowledgments

The authors would like to thank the three anonymous reviewers and Sameer Bansal, Gozde Gul Sahin, Sorcha Gilroy, Ida Szubert, Esma Balkir, Nikos Papasarantopoulos, Joana Ribeiro, Shashi Narayan, Toms Bergmanis, Clara Vania, Yang Liu and Adam Lopez for their helpful comments. This research was supported by a grant from Bloomberg and by the H2020 project SUMMA, under grant agreement 688139.