A Mention-Ranking Model for Abstract Anaphora Resolution

TL;DR

This paper introduces a mention-ranking model using an LSTM-Siamese network for abstract anaphora resolution, achieving state-of-the-art results on shell noun resolution and providing new benchmark data on the ARRAU corpus.

Contribution

It presents a novel mention-ranking approach with artificial data generation for abstract anaphora resolution and offers the first benchmark results on a challenging subset of ARRAU.

Findings

Outperforms previous methods on shell noun resolution

Model variants excel with nominal anaphors without specific training

Discriminates candidates using syntactic and deeper features

Abstract

Resolving abstract anaphora is an important, but difficult task for text understanding. Yet, with recent advances in representation learning this task becomes a more tangible aim. A central property of abstract anaphora is that it establishes a relation between the anaphor embedded in the anaphoric sentence and its (typically non-nominal) antecedent. We propose a mention-ranking model that learns how abstract anaphors relate to their antecedents with an LSTM-Siamese Net. We overcome the lack of training data by generating artificial anaphoric sentence--antecedent pairs. Our model outperforms state-of-the-art results on shell noun resolution. We also report first benchmark results on an abstract anaphora subset of the ARRAU corpus. This corpus presents a greater challenge due to a mixture of nominal and pronominal anaphors and a greater range of confounders. We found model variants that outperform the baselines for nominal anaphors, without training on individual anaphor data, but still lag behind for pronominal anaphors. Our model selects syntactically plausible candidates and -- if disregarding syntax -- discriminates candidates using deeper features.

Figures

Tables

Table 1. Table 1: S ′ ′ {}^{{}^{\prime}} -heads and the anaphoric types and phrases they induce (most frequent interpretation).

type	head of S${}^{{}^{\prime }}$	possible anaphoric phrase
empty	$\varnothing$	this, that
general	that, this	that, this
causal	because, as	therefore, because of this/that,
temporal	while, since, etc.	during this/that
conditional	if, whether	if this/that is true

Table 2. Table 2: Data statistics. For the ASN and CSN we report statistics over all shell nouns, but classifiers are trained independently.

		shell noun		abstract anaphora
		CSN train	ASN test	artifical train	ARRAU-AA test
# shell nouns / anaphors		114492	2303	8527	600
median # of tokens	Antec	12.75	13.87	11	20.5
	AnaphS	11.5	24	19	28
median #	Antec	2	4.5	2	1
	negatives	44.5	39	15	48
#	nominal	114492	2303	0	397
	pronominal	0	0	8527	203

Table 3. Table 3: Shell noun resolution results.

		s @ 1	s @ 2	s @ 3	s @ 4
fact (train: 43809, test: 472)	MR-LSTM	83.47	85.38	86.44	87.08
	KZH13	70.00	86.00	92.00	95.00
	TAGBL	46.99	-	-	-
reason (train: 4529, test: 442)	MR-LSTM	71.27	77.38	80.09	80.54
	+ tuning	87.78	91.63	93.44	93.89
	KZH13	72.00	86.90	90.00	94.00
	TAGBL	42.40	-	-	-
issue (train: 2664, test: 303)	MR-LSTM	88.12	91.09	93.07	93.40
	KZH13	47.00	61.00	72.00	81.00
	TAGBL	44.92	-	-	-
decision (train: 42289, test: 389)	MR-LSTM	76.09	85.86	91.00	93.06
	KZH13	35.00	53.00	67.00	76.00
	TAGBL	45.55	-	-	-
question (train: 9327, test: 440)	MR-LSTM	89.77	94.09	95.00	95.68
	KZH13	70.00	83.00	88.00	91.00
	TAGBL	42.02	-	-	-
possibility (train: 11874, test: 277)	MR-LSTM	93.14	94.58	95.31	95.67
	KZH13	56.00	76.00	87.00	92.00
	TAGBL	48.66	-	-	-

Table 4. Table 4: Architecture ablation for reason .

reason
ctx	aa	tag	cut	ffl1	ffl2	s@1	s@2	s@ 3	s@ 4
✓	✓	✓	✓	✓	✓	87.78	91.63	93.44	93.89
✗	✓	✓	✓	✓	✓	85.97	87.56	89.14	89.82
✓	✗	✓	✓	✓	✓	86.65	88.91	91.18	91.40
✓	✓	✗	✗	✓	✓	68.10	80.32	85.29	89.37
✓	✓	✓	✗	✓	✓	85.52	88.24	89.59	90.05
✗	✗	✗	✗	✓	✓	66.97	80.54	85.75	88.24
✓	✓	✓	✓	✗	✓	87.56	91.63	92.76	94.12
✓	✓	✓	✓	✓	✗	85.97	88.69	89.14	90.05

Table 5. Table 5: Results table for the ARRAU-AA test set. Refer to text for explanation of duplicated rows.

						all (600)				nominal (397)				pronominal (203)
ctx	aa	tag	cut	ffl1	ffl2	s@1	s@2	s@ 3	s@ 4	s@1	s@2	s@ 3	s@ 4	s@1	s@2	s@ 3	s@ 4
✓	✓	✓	✓	✓	✓	24.17	43.67	54.50	63.00	29.47	50.63	62.47	72.04	13.79	30.05	38.92	45.32
✗	✓	✓	✓	✓	✓	29.67	52.50	66.00	75.00	33.50	58.19	72.04	80.86	22.17	41.38	54.19	63.55
✓	✗	✓	✓	✓	✓	22.83	39.00	52.00	61.33	22.42	41.31	54.66	64.48	23.65	34.48	46.80	55.17
✓	✓	✗	✗	✓	✓	38.33	54.83	63.17	69.33	46.60	64.48	72.54	79.09	22.17	35.96	44.83	50.25
✓	✓	✗	✗	✓	✓	43.83	56.33	66.33	73.00	51.89	64.48	73.55	79.85	28.08	40.39	52.22	59.61
✓	✓	✗	✗	✓	✓	38.17	52.50	61.33	68.67	43.07	57.43	65.49	72.04	28.57	42.86	53.20	62.07
✓	✓	✓	✗	✓	✓	30.17	48.00	57.83	67.33	30.73	50.88	61.21	71.54	29.06	42.36	51.23	59.11
✗	✗	✗	✗	✓	✓	26.33	40.50	50.67	58.67	28.46	41.81	52.14	59.70	22.17	37.93	47.78	56.65
✓	✓	✓	✓	✗	✓	21.33	41.17	53.17	60.33	23.43	47.36	60.45	69.52	17.24	29.06	38.92	42.36
✓	✓	✓	✓	✓	✗	12.00	24.67	33.50	41.50	13.35	27.20	37.28	45.84	9.36	19.70	26.11	33.00
PSBL						27.67	-	-	-	30.48	-	-	-	22.17	-	-	-
TAGBL						38.43	-	-	-	40.10	-	-	-	35.17	-	-	-

Table 6. Table 6: HPs used for the different architecture variants for the shell noun reason .

ctx	aa	tag	cut	ffl1	ffl2	$h_{LSTM}$	$h_{ffl1}$	$h_{ffl2}$	$d_{TAG}$	$g$	$f_w$	$r$	$k_{LSTM}$	$k_{ffl1}$	$k_{ffl2}$	# param.	$t_e$	$e$
✓	✓	✓	✓	✓	✓	95	283	1115	49	2.13	9.40	${6.61}^{-5}$	0.60	0.99	0.71	1928557	3.86	9
✗	✓	✓	✓	✓	✓	140	375	1193	83	7.44	4.41	${2.87}^{-6}$	0.62	0.80	0.82	2842489	3.83	5
✓	✗	✓	✓	✓	✓	61	621	1485	81	8.27	3.27	${3.44}^{-3}$	0.56	0.94	0.99	3502713	3.71	6
✓	✓	✗	✗	✓	✓	39	722	1655	-	43.00	7.11	${2.91}^{-6}$	0.89	0.99	0.88	3624949	3.73	1
✓	✓	✓	✗	✓	✓	79	359	1454	65	3.22	7.74	${9.66}^{-6}$	0.70	0.76	0.94	2459362	4.61	2
✗	✗	✗	✗	✓	✓	38	548	1997	-	82.00	7.14	${6.07}^{-3}$	0.52	0.98	0.80	3345859	4.41	4
✓	✓	✓	✓	✗	✓	39	-	956	96	7.82	8.68	${1.64}^{-7}$	0.78	-	0.59	1567647	4.41	3
✓	✓	✓	✓	✓	✗	71	305	-	94	9.40	5.42	${8.3}^{-3}$	0.52	0.83	-	1593880	4.62	8

Equations

$$score(c,s)=W{\tilde{\mathbf{h}}}_{\mathbf{c},\mathbf{s}}+b\in \mathbb{R},$$

$$\sum _{i=1}^n\max (0,\underset{c\in \mathcal{N}(a_i)}{\max }\{1+score(c,s_i)-score(\widetilde{t_i},s_i)\}).$$

Full text

1 Introduction

Current research in anaphora (or coreference) resolution is focused on resolving noun phrases referring to concrete objects or entities in the real world, which is arguably the most frequently occurring type. Distinct from these are diverse types of abstract anaphora (AA) Asher (1993) where reference is made to propositions, facts, events or properties. An example is given in (1) below.111Example drawn from ARRAU (Uryupina et al., 2016).

While recent approaches address the resolution of selected abstract shell nouns (Kolhatkar and Hirst, 2014), we aim to resolve a wide range of abstract anaphors, such as the NP this trend in (1), as well as pronominal anaphors (this, that, or it).

Henceforth, we refer to a sentence that contains an abstract anaphor as the anaphoric sentence (AnaphS), and to a constituent that the anaphor refers to as the antecedent (Antec) (cf. (\ex1)).

\enumsentence

Ever-more powerful desktop computers, designed with one or more microprocessors as their ”brains”, are expected to increasingly take on functions carried out by more expensive minicomputers and mainframes. ”[Antec The guys that make traditional hardware are really being obsoleted by microprocessor-based machines]”, said Mr. Benton. [AnaphS As a result of this trendAA, longtime powerhouses HP, IBM and Digital Equipment Corp. are scrambling to counterattack with microprocessor-based systems of their own.]

A major obstacle for solving this task is the lack of sufficient amounts of annotated training data. We propose a method to generate large amounts of training instances covering a wide range of abstract anaphor types. This enables us to use neural methods which have shown great success in related tasks: coreference resolution Clark and Manning (2016a), textual entailment Bowman et al. (2016), learning textual similarity Mueller and Thyagarajan (2016), and discourse relation sense classification Rutherford et al. (2017).

Our model is inspired by the mention-ranking model for coreference resolution Wiseman et al. (2015); Clark and Manning (2015, 2016a, 2016b) and combines it with a Siamese Net Mueller and Thyagarajan (2016), Neculoiu et al. (2016) for learning similarity between sentences. Given an anaphoric sentence (AntecS in (1)) and a candidate antecedent (any constituent in a given context, e.g. being obsoleted by microprocessor-based machines in (1)), the LSTM-Siamese Net learns representations for the candidate and the anaphoric sentence in a shared space. These representations are combined into a joint representation used to calculate a score that characterizes the relation between them. The learned score is used to select the highest-scoring antecedent candidate for the given anaphoric sentence and hence its anaphor. We consider one anaphor at a time and provide the embedding of the context of the anaphor and the embedding of the head of the anaphoric phrase to the input to characterize each individual anaphor – similar to the encoding proposed by Zhou and Xu (2015) for individuating multiply occurring predicates in SRL. With deeper inspection we show that the model learns a relation between the anaphor in the anaphoric sentence and its antecedent. Fig. 1 displays our architecture.

In contrast to other work, our method for generating training data is not confined to specific types of anaphora such as shell nouns (Kolhatkar and Hirst, 2014) or anaphoric connectives (Stede and Grishina, 2016). It produces large amounts of instances and is easily adaptable to other languages. This enables us to build a robust, knowledge-lean model for abstract anaphora resolution that easily extends to multiple languages.

We evaluate our model on the shell noun resolution dataset of Kolhatkar et al. (2013b) and show that it outperforms their state-of-the-art results. Moreover, we report results of the model (trained on our newly constructed dataset) on unrestricted abstract anaphora instances from the ARRAU corpus Poesio and Artstein (2008); Uryupina et al. (2016). To our knowledge this provides the first state-of-the-art benchmark on this data subset.

Our TensorFlow222Abadi et al. (2015) implementation of the model and scripts for data extraction are available at: https://github.com/amarasovic/neural-abstract-anaphora.

2 Related and prior work

Abstract anaphora has been extensively studied in linguistics and shown to exhibit specific properties in terms of semantic antecedent types, their degrees of abstractness, and general discourse properties (Asher, 1993; Webber, 1991). In contrast to nominal anaphora, abstract anaphora is difficult to resolve, given that agreement and lexical match features are not applicable. Annotation of abstract anaphora is also difficult for humans (Dipper and Zinsmeister, 2012), and thus, only few smaller-scale corpora have been constructed. We evaluate our models on a subset of the ARRAU corpus (Uryupina et al., 2016) that contains abstract anaphors and the shell noun corpus used in Kolhatkar et al. (2013b).333We thank the authors for making their data available. We are not aware of other freely available abstract anaphora datasets.

Little work exists for the automatic resolution of abstract anaphora. Early work (Eckert and Strube, 2000; Strube and Müller, 2003; Byron, 2004; Müller, 2008) has focused on spoken language, which exhibits specific properties. Recently, event coreference has been addressed using feature-based classifiers Jauhar et al. (2015); Lu and Ng (2016). Event coreference is restricted to a subclass of events, and usually focuses on coreference between verb (phrase) and noun (phrase) mentions of similar abstractness levels (e.g. purchase – acquire) with no special focus on (pro)nominal anaphora. Abstract anaphora typically involves a full-fledged clausal antecedent that is referred to by a highly abstract (pro)nominal anaphor, as in (\ex0).

Rajagopal et al. (2016) proposed a model for resolution of events in biomedical text that refer to a single or multiple clauses. However, instead of selecting the correct antecedent clause(s) (our task) for a given event, their model is restricted to classifying the event into six abstract categories: this these changes, responses, analysis, context, finding, observation, based on its surrounding context. While related, their task is not comparable to the full-fledged abstract anaphora resolution task, since the events to be classified are known to be coreferent and chosen from a set of restricted abstract types.

More related to our work is Anand and Hardt (2016) who present an antecedent ranking account for sluicing using classical machine learning based on a small training dataset. They employ features modeling distance, containment, discourse structure, and – less effectively – content and lexical correlates.444Their data set was not publicized.

Closest to our work is Kolhatkar et al. (2013b) (KZH13) and Kolhatkar and Hirst (2014) (KH14) on shell noun resolution, using classical machine learning techniques. Shell nouns are abstract nouns, such as fact, possibility, or issue, which can only be interpreted jointly with their shell content (their embedded clause as in (\ex1) or antecedent as in (\ex2)). KZH13 refer to shell nouns whose antecedent occurs in the prior discourse as anaphoric shell nouns (ASNs) (cf. (\ex2)), and cataphoric shell nouns (CSNs) otherwise (cf. (\ex1)).555We follow this terminology for their approach and data representation.

\enumsentence

Congress has focused almost solely on the fact that [special education is expensive - and that it takes away money from regular education.] \enumsentenceEnvironmental Defense […] notes that [Antec “Mowing the lawn with a gas mower produces as much pollution […] as driving a car 172 miles.“] [AnaphS This fact may […] explain the recent surge in the sales of […] old-fashioned push mowers […]].

KZH13 presented an approach for resolving six typical shell nouns following the observation that CSNs are easy to resolve based on their syntactic structure alone, and the assumption that ASNs share linguistic properties with their embedded (CSN) counterparts. They manually developed rules to identify the embedded clause (i.e. cataphoric antecedent) of CSNs and trained SVMrank Joachims (2002) on such instances. The trained SVMrank model is then used to resolve ASNs. KH14 generalized their method to be able to create training data for any given shell noun, however, their method heavily exploits the specific properties of shell nouns and does not apply to other types of abstract anaphora.

Stede and Grishina (2016) study a related phenomenon for German. They examine inherently anaphoric connectives (such as demzufolge – according to which) that could be used to access their abstract antecedent in the immediate context. Yet, such connectives are restricted in type, and the study shows that such connectives are often ambiguous with nominal anaphors and require sense disambiguation. We conclude that they cannot be easily used to acquire antecedents automatically.

In our work, we explore a different direction: we construct artificial training data using a general pattern that identifies embedded sentence constituents, which allows us to extract relatively secure training data for abstract anaphora that captures a wide range of anaphora-antecedent relations, and apply this data to train a model for the resolution of unconstrained abstract anaphora.

Recent work in entity coreference resolution has proposed powerful neural network-based models that we will adapt to the task of abstract anaphora resolution. Most relevant for our task is the mention-ranking neural coreference model proposed in Clark and Manning (2015), and their improved model in Clark and Manning (2016a), which integrates a loss function (Wiseman et al., 2015) which learns distinct feature representations for anaphoricity detection and antecedent ranking.

Siamese Nets distinguish between similar and dissimilar pairs of samples by optimizing a loss over the metric induced by the representations. It is widely used in vision Chopra et al. (2005), and in NLP for semantic similarity, entailment, query normalization and QA (Mueller and Thyagarajan, 2016; Neculoiu et al., 2016; Das et al., 2016).

3 Mention-Ranking Model

Given an anaphoric sentence $s$ with a marked anaphor (mention) and a candidate antecedent $c$, the mention-ranking (MR) model assigns the pair $(c,s)$ a score, using representations produced by an LSTM-Siamese Net. The highest-scoring candidate is assigned to the marked anaphor in the anaphoric sentence. Fig. 1 displays the model.

We learn representations of an anaphoric sentence $s$ and a candidate antecedent $c$ using a bi-directional Long Short-Term Memory Hochreiter and Schmidhuber (1997); Graves and Schmidhuber (2005). One bi-LSTM is applied to the anaphoric sentence $s$ and a candidate antecedent $c$, hence the term siamese. Each word is represented with a vector $\boldsymbol{w_{i}}$ constructed by concatenating embeddings of the word, of the context of the anaphor (average of embeddings of the anaphoric phrase, the previous and the next word), of the head of the anaphoric phrase666Henceforth we refer to it as embedding of the anaphor., and, finally, an embedding of the constituent tag of the candidate, or the S constituent tag if the word is in the anaphoric sentence. For each sequence $s$ or $c$, the word vectors $\boldsymbol{w_{i}}$ are sequentially fed into the bi-LSTM, which produces outputs from the forward pass, $\overrightarrow{\boldsymbol{h_{i}}}$, and outputs $\boldsymbol{\overleftarrow{h_{i}}}$ from the backward pass. The final output of the i-th word is defined as $\boldsymbol{h_{i}}=[\boldsymbol{\overleftarrow{h_{i}}};\boldsymbol{\overrightarrow{h_{i}}}]$. To get a representation of the full sequence, $\boldsymbol{h_{s}}$ or $\boldsymbol{h_{c}}$, all outputs are averaged, except for those that correspond to padding tokens.

To prevent forgetting the constituent tag of the sequence, we concatenate the corresponding tag embedding with $\boldsymbol{h_{s}}$ or $\boldsymbol{h_{c}}$ (we call this a shortcut for the tag information). The resulting vector is fed into a feed-forward layer of exponential linear units (ELUs) Clevert et al. (2016) to produce the final representation $\boldsymbol{\tilde{h}_{s}}$ or $\boldsymbol{\tilde{h}_{c}}$ of the sequence.

From $\boldsymbol{\tilde{h}_{c}}$ and $\boldsymbol{\tilde{h}_{s}}$ we compute a vector $\boldsymbol{h_{c,s}}=[|\boldsymbol{\tilde{h}_{c}}-\boldsymbol{\tilde{h}_{s}}|;\boldsymbol{\tilde{h}_{c}}\odot\boldsymbol{\tilde{h}_{s}}]$ Tai et al. (2015), where $|\text{--}|$ denotes the absolute values of the element-wise subtraction, and $\odot$ the element-wise multiplication. Then $\boldsymbol{h_{c,s}}$ is fed into a feed-forward layer of ELUs to obtain the final joint representation, $\boldsymbol{\tilde{h}_{c,s}}$, of the pair $(c,s)$. Finally, we compute the score for the pair $(c,s)$ that represents relatedness between them, by applying a single fully connected linear layer to the joint representation:

[TABLE]

where W is a $1\times d$ weight matrix, and $d$ the dimension of the vector $\boldsymbol{\tilde{h}_{c,s}}$.

We train the described mention-ranking model with the max-margin training objective from Wiseman et al. (2015), used for the antecedent ranking subtask. Suppose that the training set $\mathcal{D}=\{(a_{i},s_{i},\mathcal{T}(a_{i}),\mathcal{N}(a_{i})\}_{i=1}^{n}$, where $a_{i}$ is the i-th abstract anaphor, $s_{i}$ the corresponding anaphoric sentence, \mathcal{T}$$(a_{i}) the set of antecedents of $a_{i}$ and \mathcal{N}$$(a_{i}) the set of candidates that are not antecedents (negative candidates). Let $\tilde{t_{i}}=\operatorname*{arg\,max}_{t\in\mathcal{T}(a_{i})}score(t_{i},s_{i})$ be the highest scoring antecedent of $a_{i}$. Then the loss is given by

[TABLE]

4 Training data construction

We create large-scale training data for abstract anaphora resolution by exploiting a common construction, consisting of a verb with an embedded sentence (complement or adverbial) (cf. Fig. 2). We detect this pattern in a parsed corpus, ’cut off’ the S*′* constituent and replace it with a suitable anaphor to create the anaphoric sentence (AnaphS), while S yields the antecedent (Antec). This method covers a wide range of anaphora-antecedent constellations, due to diverse semantic or discourse relations that hold between the clause hosting the verb and the embedded sentence.

First, the pattern applies to verbs that embed sentential arguments. In (\ex1), the verb doubt establishes a specific semantic relation between the embedding sentence and its sentential complement.

\enumsentence

He doubts [${}_{S^{\prime}}$[S a Bismarckian super state will emerge that would dominate Europe], but warns of ”a risk of profound change in the [..] European Community from a Germany that is too strong, even if democratic”].

From this we extract the artificial antecedent A Bismarckian super state will emerge that would dominate Europe, and its corresponding anaphoric sentence He doubts this, but warns of ”a risk of profound change … even if democratic”, which we construct by randomly choosing one of a pre-defined set of appropriate anaphors (here: this, that, it), cf. Table 1. The second row in Table 1 is used when the head of S*′* is filled by an overt complementizer (doubts that), as opposed to (\ex0). The remaining rows in Table 1 apply to adverbial clauses of different types.

Adverbial clauses encode specific discourse relations with their embedding sentences, often indicated by their conjunctions. In (\ex1), for example, the causal conjunction as relates a cause (embedded sentence) and its effect (embedding sentence):

\enumsentence

There is speculation that property casualty firms will sell even more munis [${}_{S^{\prime}}$ as [S they scramble to raise cash to pay claims related to Hurricane Hugo [..] ]].

We randomly replace causal conjunctions because, as with appropriately adjusted anaphors, e.g. because of that, due to this or therefore that make the causal relation explicit in the anaphor.777In case of ambiguous conjunctions (e.g. as interpreted as causal or temporal), we generally choose the most frequent interpretation.

Compared to the shell noun corpus of KZH13, who made use of a carefully constructed set of extraction patterns, a downside of our method is that our artificially created antecedents are uniformly of type S. However, the majority of abstract anaphora antecedents found in the existing datasets are of type S. Also, our models are intended to induce semantic representations, and so we expect syntactic form to be less critical, compared to a feature-based model.888This also alleviates problems with languages like German, where (non-)embedded sentences differ in surface position of the finite verb. We can either adapt the order or ignore it, when producing anaphoric sentence – antecedent pairs. Finally, the general extraction pattern in Fig. 2, covers a much wider range of anaphoric types.

Using this method we generated a dataset of artificial anaphoric sentence–antecedent pairs from the WSJ part of the PTB Corpus (Marcus et al., 1993), automatically parsed using the Stanford Parser Klein and Manning (2003).

5 Experimental setup

5.1 Datasets

We evaluate our model on two types of anaphora: (a) shell noun anaphora and (b) (pro)nominal abstract anaphors extracted from ARRAU.

a. Shell noun resolution dataset.

For comparability we train and evaluate our model for shell noun resolution, using the original training (CSN) and test (ASN) corpus of Kolhatkar et al. (2013a, b).999We thank the authors for providing the available data. We follow the data preparation and evaluation protocol of Kolhatkar et al. (2013b) (KZH13).

The CSN corpus was constructed from the NYT corpus using manually developed patterns to identify the antecedent of cataphoric shell nouns (CSNs). In KZH13, all syntactic constituents of the sentence that contains both the CSN and its antecedent were considered as candidates for training a ranking model. Candidates that differ from the antecedent in only one word or one word and punctuation were as well considered as antecedents101010We obtained this information from the authors directly.. To all other candidates we refer to as negative candidates. For every shell noun, KZH13 used the corresponding part of the CSN data to train SVMrank.

The ASN corpus serves as the test corpus. It was also constructed from the NYT corpus, by selecting anaphoric instances with the pattern ”this $\langle\textit{shell noun}\rangle$” for all covered shell nouns. For validation, Kolhatkar et al. (2013a) crowdsourced annotations for the sentence which contains the antecedent, which KZH13 refer to as a broad region. Candidates for the antecedent were obtained by using all syntactic constituents of the broad region as candidates and ranking them using the SVMrank model trained on the CSN corpus. The top 10 ranked candidates were presented to the crowd workers and they chose the best answer that represents the ASN antecedent. The workers were encouraged to select None when they did not agree with any of the displayed answers and could provide information about how satisfied they were with the displayed candidates. We consider this dataset as gold, as do KZH13, although it may be biased towards the offered candidates.111111The authors provided us with the workers’ annotations of the broad region, antecedents chosen by the workers and links to the NYT corpus. The extraction of the anaphoric sentence and the candidates had to be redone.

b. Abstract anaphora resolution data set.

We use the automatically constructed data from the WSJ corpus (Section 4) for training.121212We excluded any documents that are part of ARRAU. Our test data for unrestricted abstract anaphora resolution is obtained from the ARRAU corpus (Uryupina et al., 2016). We extracted all abstract anaphoric instances from the WSJ part of ARRAU that are marked with the category abstract or plan,131313ARRAU distinguishes abstract anaphors and (mostly) pronominal anaphors referring to an action or plan, as plan. and call the subcorpus ARRAU-AA.

Candidates extraction.

Following KZH13, for every anaphor we create a list of candidates by extracting all syntactic constituents from sentences which contain antecedents. Candidates that differ from antecedents in only one word, or one word and punctuation, were as well considered as antecedents. Constituents that are not antecedents are considered as negative candidates.

Data statistics.

Table 2 gives statistics of the datasets: the number of anaphors (row 1), the median length (in tokens) of antecedents (row 2), the median length (in tokens) for all anaphoric sentences (row 3), the median of the number of antecedents and candidates that are not antecedents (negatives) (rows 4–5), the number of pronominal and nominal anaphors (rows 6–7). Both training sets, artificial and CSN, have only one possible antecedent for which we accept two minimal variants differing in only one word or one word and punctuation. On the contrary, both test sets by design allow annotation of more than one antecedent that differ in more than one word. Every anaphor in the artificial training dataset is pronominal, whereas anaphors in CSN and ASN are nominal only. ARRAU-AA has a mixture of nominal and pronominal anaphors.

Data pre-processing.

Other details can be found in Supplementary Materials.

5.2 Baselines and evaluation metrics

Following KZH13, we report success@n (s@n), which measures whether the antecedent, or a candidate that differs in one word141414We obtained this information in personal communication with one of the authors., is in the first $n$ ranked candidates, for $n\in\{1,2,3,4\}$. Additionally, we report the preceding sentence baseline (PSBL) that chooses the previous sentence for the antecedent and TAGbaseline (TAGBL) that randomly chooses a candidate with the constituent tag label in $\{$S, VP, ROOT, SBAR$\}$. For TAGBL we report the average of 10 runs with 10 fixed seeds. PSBL always performs worse than the KZH13 model on the ASN, so we report it only for ARRAU-AA.

5.3 Training details for our models

Hyperparameters tuning. We recorded performance with manually chosen HPs and then tuned HPs with Tree-structured Parzen Estimators (TPE) Bergstra et al. (2011)151515https://github.com/hyperopt/hyperopt.. TPE chooses HPs for the next (out of $10$) trails on the basis of the *s@*1 score on the devset. As devsets we employ the ARRAU-AA corpus for shell noun resolution and the ASN corpus for unrestricted abstract anaphora resolution. For each trial we record performance on the test set. We report the best test *s@*1 score in $10$ trials if it is better than the scores from default HPs. The default HPs and prior distributions for HPs used by TPE are given below. The (exact) HPs we used can be found in Supplementary Materials.

Input representation. To construct word vectors $\boldsymbol{w_{i}}$ as defined in Section 3, we used $100$-dim. GloVe word embeddings pre-trained on the Gigaword and Wikipedia Pennington et al. (2014), and did not fine-tune them. Vocabulary was built from the words in the training data with frequency in $\{3,\text{ }\mathcal{U}(1,10)\}$, and OOV words were replaced with an UNK token. Embeddings for tags are initialized with values drawn from the uniform distribution \mathcal{U}\big{(}-\frac{1}{\sqrt{d+t}},\frac{1}{\sqrt{d+t}}\big{)}, where $t$ is the number of tags161616We used a list of tags obtained from the Stanford Parser. and $d\in\{50,\textit{qlog-}\mathcal{U}(30,100)\}$ the size of the tag embeddings.171717qlog-$\mathcal{U}$ is the so-called qlog-uniform distribution. We experimented with removing embeddings for tag, anaphor and context.

Weights initialization. The size of the LSTMs hidden states was set to $\{100,$ $\textit{qlog-}\mathcal{U}(30,150)\}$. We initialized the weight matrices of the LSTMs with random orthogonal matrices Henaff et al. (2016), all other weight matrices with the initialization proposed in He et al. (2015). The first feed-forward layer size is set to a value in $\{400,\textit{qlog-}\mathcal{U}(200,800)\}$, the second to a value in $\{1024,\textit{qlog-}\mathcal{U}(400,2000)\}$. Forget biases in the LSTM were initialized with 1s Józefowicz et al. (2015), all other biases with 0s.

Optimization. We trained our model in mini-batches using Adam Kingma and Ba (2015) with the learning rate of $10^{-4}$ and maximal batch size $64$. We clip gradients by global norm Pascanu et al. (2013), with a clipping value in $\{1.0,\text{ }\mathcal{U}(1,100)\}$. We train for $10$ epochs and choose the model that performs best on the devset.

Regularization. We used the $l_{2}$-regularization with $\lambda\in\{10^{-5},\textit{ log-}\mathcal{U}(10^{-7},10^{-2})\}$. Dropout Srivastava et al. (2014) with a keep probability $k_{p}\in\{0.8,\text{ }\mathcal{U}(0.5,1.0)\}$ was applied to the outputs of the LSTMs, both feed-forward layers and optionally to the input with $k_{p}\in\mathcal{U}(0.8,1.0)$.

6 Results and analysis

6.1 Results on shell noun resolution dataset

Table 3 provides the results of the mention-ranking model (MR-LSTM) on the ASN corpus using default HPs. Column 2 states which model produced the results: KZH13 refers to the best reported results in Kolhatkar et al. (2013b) and TAGBL is the baseline described in Section 5.2.

In terms of s@1 score, MR-LSTM outperforms both KZH13’s results and TAGBL without even necessitating HP tuning. For the outlier reason we tuned HPs (on ARRAU-AA) for different variants of the architecture: the full architecture, without embedding of the context of the anaphor (ctx), of the anaphor (aa), of both constituent tag embedding and shortcut (tag,cut), dropping only the shortcut (cut), using only word embeddings as input (ctx,aa,tag,cut), without the first (ffl1) and second (ffl2) layer. From Table 4 we observe: (1) with HPs tuned on ARRAU-AA, we obtain results well beyond KZH13, (2) all ablated model variants perform worse than the full model, (3) a large performance drop when omitting syntactic information (tag,cut) suggests that the model makes good use of it. However, this could also be due to a bias in the tag distribution, given that all candidates stem from the single sentence that contains antecedents. The median occurrence of the S tag among both antecedents and negative candidates is $1$, thus the model could achieve $50.00$ s@1 by picking S-type constituents, just as TAGBL achieves $42.02$ for reason and $48.66$ for possibility.

Tuning of HPs gives us insight into how different model variants cope with the task. For example, without tuning the model with and without syntactic information achieves $71.27$ and $19.68$ (not shown in table) *s@*1 score, respectively, and with tuning: $87.78$ and $68.10$. Performance of $68.10$ *s@*1 score indicates that the model is able to learn without syntactic guidance, contrary to the $19.68$ *s@*1 score before tuning.

6.2 Results on the ARRAU corpus

Table 5 shows the performance of different variants of the MR-LSTM with HPs tuned on the ASN corpus (always better than the default HPs), when evaluated on $3$ different subparts of the ARRAU-AA: all $600$ abstract anaphors, $397$ nominal and $203$ pronominal ones. HPs were tuned on the ASN corpus for every variant separately, without shuffling of the training data. For the best performing variant, without syntactic information (tag,cut), we report the results with HPs that yielded the best *s@*1 test score for all anaphors (row 4), when training with those HPs on shuffled training data (row 5), and with HPs that yielded the best *s@*1 score for pronominal anaphors (row 6).

The MR-LSTM is more successful in resolving nominal than pronominal anaphors, although the training data provides only pronominal ones. This indicates that resolving pronominal abstract anaphora is harder compared to nominal abstract anaphora, such as shell nouns. Moreover, for shell noun resolution in KZH13’s dataset, the MR-LSTM achieved *s@*1 scores in the range $76.09$–$93.14$, while the best variant of the model achieves $51.89$ *s@*1 score for nominal anaphors in ARRAU-AA. Although lower performance is expected, since we do not have specific training data for individual nominals in ARRAU-AA, we suspect that the reason for better performance for shell noun resolution in KZH13 is due to a larger number of positive candidates in ASN (cf. Table 2, rows: antecedents/negatives).

We also note that HPs that yield good performance for resolving nominal anaphors are not necessarily good for pronominal ones (cf. rows 4–6 in Table 5). Since the TPE tuner was tuned on the nominal-only ASN data, this suggest that it would be better to tune HPs for pronominal anaphors on a different dataset or stripping the nouns in ASN.

Contrary to shell noun resolution, omitting syntactic information boosts performance in ARRAU-AA. We conclude that when the model is provided with syntactic information, it learns to pick S-type candidates, but does not continue to learn deeper features to further distinguish them or needs more data to do so. Thus, the model is not able to point to exactly one antecedent, resulting in a lower *s@*1 score, but does well in picking a few good candidates, which yields good *s@*2-4 scores. This is what we can observe from row 2 vs. row 6 in Table 5: the MR-LSTM without context embedding (ctx) achieves a comparable *s@*2 score with the variant that omits syntactic information, but better s@3-4 scores. Further, median occurrence of tags not in {S, VP, ROOT, SBAR} among top-4 ranked candidates is [math] for the full architecture, and $1$ when syntactic information is omitted. The need for discriminating capacity of the model is more emphasized in ARRAU-AA, given that the median occurrence of S-type candidates among negatives is $2$ for nominal and even $3$ for pronominal anaphors, whereas it is $1$ for ASN. This is in line with the lower TAGBL in ARRAU-AA.

Finally, not all parts of the architecture contribute to system performance, contrary to what is observed for reason. For nominal anaphors, the anaphor (aa) and feed-forward layers (ffl1, ffl2) are beneficial, for pronominals only the second ffl.

6.3 Exploring the model

We finally analyze deeper aspects of the model: (1) whether a learned representation between the anaphoric sentence and an antecedent establishes a relation between a specific anaphor we want to resolve and the antecedent and (2) whether the max-margin objective enforces a separation of the joint representations in the shared space.

(1) We claim that by providing embeddings of both the anaphor and the sentence containing the anaphor we ensure that the learned relation between antecedent and anaphoric sentence is dependent on the anaphor under consideration. Fig. 3 illustrates the heatmap for an anaphoric sentence with two anaphors. The i-th column of the heatmap corresponds to absolute differences between the output of the bi-LSTM for the i-th word in the anaphoric sentence when the first vs. second anaphor is resolved. Stronger color indicates larger difference, the blue rectangle represents the column for the head of the first anaphor, the dashed blue rectangle the column for the head of the second anaphor. Clearly, the representations differ when the first vs.  second anaphor is being resolved and consequently, joint representations with an antecedent will differ too.

(2) It is known that the max-margin objective separates the best-scoring positive candidate from the best-scoring negative candidate. To investigate what the objective accomplishes in the MR-LSTM model, we analyze the joint representations of candidates and the anaphoric sentence (i.e., outputs of ffl2) after training. For a randomly chosen instance from ARRAU-AA, we plotted outputs of ffl2 with the tSNE algorithm v.d. Maaten and Hinton (2008). Fig. 4 illustrates that the joint representation of the first ranked candidate and the anaphoric sentence is clearly separated from other joint representations. This shows that the max-margin objective separates the best scoring positive candidate from the best scoring negative candidate by separating their respective joint representations with the anaphoric sentence.

7 Conclusions

We presented a neural mention-ranking model for the resolution of unconstrained abstract anaphora, and applied it to two datasets with different types of abstract anaphora: the shell noun dataset and a subpart of ARRAU with (pro)nominal abstract anaphora of any type. To our knowledge this work is the first to address the unrestricted abstract anaphora resolution task with a neural network. Our model also outperforms state-of-the-art results on the shell noun dataset.

In this work we explored the use of purely artificially created training data and how far it can bring us. In future work, we plan to investigate mixtures of (more) artificial and natural data from different sources (e.g. ASN, CSN).

On the more challenging ARRAU-AA, we found model variants that surpass the baselines for the entire and the nominal part of ARRAU-AA, although we do not train models on individual (nominal) anaphor training data like the related work for shell noun resolution. However, our model still lags behind for pronominal anaphors. Our results suggest that models for nominal and pronominal anaphors should be learned independently, starting with tuning of HPs on a more suitable devset for pronominal anaphors.

We show that the model can exploit syntactic information to select plausible candidates, but that when it does so, it does not learn how to distinguish candidates of equal syntactic type. By contrast, if the model is not provided with syntactic information, it learns deeper features that enable it to pick the correct antecedent without narrowing down the choice of candidates. Thus, in order to improve performance, the model should be enforced to first select reasonable candidates and then continue to learn features to distinguish them, using a larger training set that is easy to provide.

In future work we will design such a model, and offer it candidates chosen not only from sentences containing the antecedent, but the larger context.

Acknowledgments

This work has been supported by the German Research Foundation as part of the Research Training Group Adaptive Preparation of Information from Heterogeneous Sources (AIPHES) under grant No. GRK 1994/1. We would like to thank anonymous reviewers for useful comments and especially thank Todor Mihaylov for the model implementations advices and everyone in the Computational Linguistics Group for helpful discussion.

Appendix A Pre-processing details

The CSN corpus we obtained from the authors contained tokenized sentences for antecedents and anaphoric sentences. The number of instances differed from the reported numbers in KZH13 in $9$ to $809$ instances for training, and $1$ for testing. The given sentences still contained the antecedent, so we removed it from the sentence and transformed the corresponding shell into ”this $\langle\textit{shell noun}\rangle$”. An example of this process is: The decision to disconnect the ventilator came after doctors found no brain activity. $\rightarrow$ This decision came after doctors found no brain activity.

To use pre-trained word embeddings we had to lowercase all the data. As we use an automatic parse to extract all syntactic constituents, due to parser errors, candidates with the same string appeared with different tags. We eliminated duplicates by checking which tag is more frequent for candidates which have the same POS tag of the first word as the duplicated candidate, in the whole dataset. In case duplicated candidates were still occurring, we chose any of them. If such duplicates occur in antecedents, we don’t take such instances in the training data to eliminate noise, or choose any of them for the test data. For the training data we choose instances with an anaphoric sentence length of at least $10$ tokens.

All sentences in the batch are padded with a PAD token up to the maximal sentence length in the batch and corresponding hidden states in the LSTM are masked with zeros. To implement the model efficiently in TensorFlow, batches are constructed in such a way that every sentence instance in the batch has the same number of positive candidates and the same number of negative candidates. Note that by this we do not mean that the ratio of positive and negative examples is 1:1.

Appendix B Hyperparameter details

Tables 7 and 7 report the tuned HPs for resolution of the shell noun reason and resolution of abstract anaphors in ARRAU-AA for different model variants. Below is the list of all tunable HPs.

• •

the dimensionality of the hidden states in the bi-LSTM, $h_{LSTM}$

• •

the first feed-forward layer size, $h_{ffl1}$

• •

the second feed-forward layer size, $h_{ffl2}$

• •

the dimensionality of the tag embeddings, $d_{TAG}$

• •

gradient clipping value, $g$

• •

frequency of words in vocabulary, $f_{w}$

• •

regularization coefficient, $r$

• •

keep probability of outputs of bi-LSTM, $k_{LSTM}$

• •

keep probability of input, $k_{input}$

• •

keep probability of outputs of the first feed-forward layer, $k_{ffl1}$

• •

keep probability of second of the first feed-forward layer, $k_{ffl2}$

We additionally report the number of trainable parameters (# param)̇, the average epoch training time using one Nvidia GeForce GTX1080 gpu ($t_{e}$) and the epoch after which the best score is achieved ($e$).