Neural Mention Detection

TL;DR

This paper introduces and compares three neural network models for mention detection, demonstrating significant improvements in recall and F1 scores over existing methods, and enhancing coreference resolution and nested NER tasks.

Contribution

The paper presents a novel neural mention detection approach using BERT and biaffine classifiers, achieving state-of-the-art results across multiple datasets.

Findings

Best model improves mention recall by up to 1.8 percentage points.

Model achieves up to 6.2 percentage points higher F1 on CONLL and CRAC datasets.

Enhanced coreference resolution performance with up to 1.7 percentage points improvement.

Abstract

Mention detection is an important preprocessing step for annotation and interpretation in applications such as NER and coreference resolution, but few stand-alone neural models have been proposed able to handle the full range of mentions. In this work, we propose and compare three neural network-based approaches to mention detection. The first approach is based on the mention detection part of a state of the art coreference resolution system; the second uses ELMO embeddings together with a bidirectional LSTM and a biaffine classifier; the third approach uses the recently introduced BERT model. Our best model (using a biaffine classifier) achieves gains of up to 1.8 percentage points on mention recall when compared with a strong baseline in a HIGH RECALL coreference annotation setting. The same model achieves improvements of up to 5.3 and 6.2 p.p. when compared with the best-reported mention detection F1 on the CONLL and CRAC coreference data sets respectively in a HIGH F1 annotation setting. We then evaluate our models for coreference resolution by using mentions predicted by our best model in start-of-the-art coreference systems. The enhanced model achieved absolute improvements of up to 1.7 and 0.7 p.p. when compared with our strong baseline systems (pipeline system and end-to-end system) respectively. For nested NER, the evaluation of our model on the GENIA corpora shows that our model matches or outperforms state-of-the-art models despite not being specifically designed for this task.

Figures

Tables

Table 1. Table 1: Major hyperparameters for our models. Lee , Bia , Ber are used to indicate Lee md , Biaffine md , Bert md respectively.

Model	Parameter	Value
Lee, Bia	BiLSTM layers	3
Lee, Bia	BiLSTM size	200
Lee, Bia	BiLSTM dropout	0.4
Ber	Transformer layers	12
Ber	Transformer size	768
Ber	Transformer dropout	0.1
Lee, Bia, Ber	FFNN layers	2
Lee, Bia, Ber	FFNN size	150
Lee, Bia, Ber	FFNN dropout	0.2
Lee, Bia, Ber	Embeddings dropout	0.5
Lee, Bia, Ber	Optimiser	Adam
Lee, Bia	Learning rate	1e-3
Ber	Learning rate	2e-5
Lee, Bia, Ber	Training step	40K

Table 2. Table 2: Performance comparison between our mention detectors and the baseline ( ? ) system) in a high recall setting.

Data	Model	R	P	F1
conll	?)	96.6	28.2	43.7
	Lee md	97.3	28.4	44.0
	Biaffine md	97.5	28.5	44.1
	Bert md	97.3	28.4	44.0
crac3	?)	95.4	34.4	50.6
	Lee md	96.9	35.0	51.4
	Biaffine md	97.2	35.0	51.5
	Bert md	96.2	34.7	51.0

Table 3. Table 3: Comparison between our Biaffine md and the top performing systems on the mention detection task using the conll and crac data sets.

Data	Model	R	P	F1
conll	?)	89.5	40.4	55.7
	?)	74.0	73.5	73.8
	high recall	97.5	28.5	44.1
	high f1	76.0	82.6	79.1
	balance	90.6	53.0	66.9
crac3	?)	67.3	71.6	69.4
	?)	86.2	79.3	82.6
	high recall	95.3	74.2	83.5
	high f1	87.9	89.7	88.8

Table 4. Table 4: Model complexity and inference speed (docs/s) comparison between our mention detectors

	Num of	Infer Speed
Model	Parameters	conll	crac
Lee md	4 M	6.8	5.5
Biaffine md	3.4 M	8.3	6.7
Bert md	110 M	3.3	4.3

Table 5. Table 5: Comparison between the baselines and the models enhanced by our Biaffine md on the coreference resolution task.

Data	Model	MUC			B3			CEAF${}_{{\varphi }_4}$			Avg. F1
		P	R	F1	P	R	F1	P	R	F1
conll	?)	81.4	79.5	80.4	72.2	69.5	70.8	68.2	67.1	67.6	73.0
	+ high recall	80.0	79.5	79.7	70.5	69.5	70.0	67.3	66.9	67.1	72.3
	+ high recall + joint	80.9	79.2	80.0	72.0	69.0	70.5	67.7	66.9	67.3	72.6
	?)	78.4	73.4	75.8	68.6	61.8	65.0	62.7	59.0	60.8	67.2
	+ high recall	78.6	74.0	76.2	68.9	62.2	65.4	63.2	59.6	61.4	67.7
	?)	79.2	70.4	74.6	69.9	58.0	63.4	63.5	55.5	59.2	65.7
	+ high recall	78.7	72.4	75.4	69.4	59.7	64.2	62.2	57.7	59.9	66.5
	+ balance	80.3	72.5	76.2	71.2	60.4	65.3	64.6	57.1	60.6	67.4
crac	?)	79.2	71.9	75.3	72.4	63.5	67.7	66.2	58.6	62.2	68.4
	+ high recall	76.2	73.1	74.6	68.4	65.5	66.9	65.1	61.8	63.4	68.3
	+ high recall + joint	77.6	73.4	75.4	70.4	65.5	67.9	66.4	61.9	64.1	69.1

Table 6. Table 6: Overall performance comparison on genia81 test set.

Model	R	P	F1
?)	73.6	75.9	74.7
?)	64.0	93.2	77.1
Our model	78.5	77.8	78.2

Table 7. Table 7: Individual category performance comparison on genia81 test set. Zh refers to ? ) and So refers to ? )

Categories	R	P	F1	So F1	Zh F1
DNA	74.0	75.3	74.6	71.8	70.6
RNA	85.3	84.5	84.9	72.4	81.5
protein	82.6	78.5	80.5	80.8	76.4
cell line	70.1	77.4	73.6	67.9	71.3
cell type	72.1	78.3	75.1	78.1	72.5

Table 8. Table 8: Overall performance comparison on genia90 test set.

Model	R	P	F1
?)	71.3	78.5	74.7
?)	73.9	75.8	74.8
?)	-	-	78.3
Our model	78.0	79.1	78.6

Table 9. Table 9: Individual category performance comparison on genia90 test set. Ju refers to ? )

Categories	R	P	F1	Ju F1
DNA	72.4	78.9	75.5	72.0
RNA	88.1	86.5	87.3	84.5
protein	82.2	79.5	80.8	76.7
cell line	67.4	80.0	73.2	71.2
cell type	74.4	75.4	74.9	72.0

Equations

$${\alpha }_t=\text{textsc}{FFNN}_{\alpha }(x_t^{\ast })$$

$$a_{i,t}=\frac{exp({\alpha }_t)}{{\sum }_{k=s_i}^{e_i}exp({\alpha }_k)}$$

$$h_i^{\ast }=\sum _{t=s_i}^{e_i}{a}_{i,t}\cdot x_t$$

$$N_i^{\ast }=[x_{s_i}^{\ast },x_{e_i}^{\ast },h_i^{\ast },\varphi (i)]$$

$$r_m(i)=\text{textsc}{FFNN}_m(N_i^{\ast })$$

$$p_m(i)=\frac{1}{1+e^{-r_m(i)}}$$

$$h_s(i)=\text{textsc}{FFNN}_s(x_{s_i}^{\ast })$$

$$h_e(i)=\text{textsc}{FFNN}_e(x_{e_i}^{\ast })$$

$$r_m(i)=h_s(i{)}^{\top }{W}_m{h}_e(i)+h_s(i)b_m$$

$$p_{m}(i)=\Bigg{\{}\begin{tabular}[]{ll}$\frac{1}{1+e^{-r_{m}(i)}}$&$s_{i}\leq e_{i}$\\ $0$&$s_{i}>e_{i}$\end{tabular}$$

$$N_i^{\ast }=[x_{s_i}^{\ast },x_{e_i}^{\ast }]$$

$$r_m(i)=\text{textsc}{FFNN}_m(N_i^{\ast })$$

$$p_m(i)=\frac{1}{1+e^{-r_m(i)}}$$

$$p_m(i_c)=\frac{e^{r_m(i_c)}}{{\sum }_{\hat{c}=1}^C{e}^{r_m(i_{\hat{c}})}}$$

$$-\sum _{i=1}^N{y}_i\log p_m(i)+(1-y_i)\log (1-p_m(i))$$

$$-\sum _{i=1}^N\sum _{c=1}^C{y}_{i_c}\log p_m(i_c)$$

Full text

**Keywords: **Mention Detection, Coreference Resolution, Nested Named Entity Recognition, Deep Neural Networks

Neural Mention Detection

[TABLE]

Abstract content

1. Introduction

Mention detection (md) is the task of identifying mentions of entities in text. It is an important preprocessing step for downstream applications such as nested named entity recognition [Zheng et al., 2019] or coreference resolution [Poesio et al., 2016]; thus, the quality of mention detection affects both the performance of models for such applications and the quality of annotated data used to train them [Chamberlain et al., 2016, Poesio et al., 2019].

Much mention detection research for ner has concentrated on a simplified version of md that focuses on proper names only (i.e., it doesn’t consider as mentions nominals such as the protein or pronouns such as it), and ignores the fact that mentions may nest (e.g., noun phrases such as [[CCITA] mRNA] in the genia corpus are mentions of two separate entities, CCITA and CCITA and mRNA [Alex et al., 2007]). However such simplified view of mentions is not sufficient for ner in domains such as biomedical, or for coreference, that requires full mention detection. Another limitation of typical mention detection systems is that they only predict mentions in a high f1 fashion, whereas in coreference, for instance, mentions are usually predicted in a high recall setting, since further pruning will be carried out at the coreference system [Clark and Manning, 2016b, Clark and Manning, 2016a, Lee et al., 2017, Lee et al., 2018, Kantor and Globerson, 2019].

There are only very few recent studies that attempt to apply neural network approaches to develop a standalone mention detector. Neural network approaches using context-sensitive embeddings such as elmo [Peters et al., 2018] and bert [Devlin et al., 2019] have resulted in substantial improvements for mention detectors in the ner benchmark conll 2003 data set. However, most coreference systems that appeared after Lee et al., [Lee et al., 2017, Lee et al., 2018] carry out mention detection as a part of their end-to-end coreference system. Such systems do not output intermediate mentions, hence the mention detector cannot be directly used to extract mentions for an annotation project, or by other coreference systems. Thus the only standalone mention detectors that can be used as preprocessing for a coreference annotation are ones that do not take advantage of these advances and still heavily rely on parsing to identify all nps as candidate mentions [Björkelund and Kuhn, 2014, Wiseman et al., 2015, Wiseman et al., 2016] or ones that use the rule-based mention detector from the Stanford deterministic system [Lee et al., 2013] to extract mentions from nps, named entity mentions and pronouns [Clark and Manning, 2015, Clark and Manning, 2016b]. To the best of our knowledge, ?) introduced the only standalone neural mention detector. By using a modified version of the ner system of ?), they showed substantial performance gains at mention detection on the benchmark conll 2012 data set and on the crac 2018 data set when compared with the Stanford deterministic system.

In this paper, we compare three neural architectures for standalone md. The first system is a slightly modified version of the mention detection part of the ?) system. The second system employs a bi-directional lstm on the sentence level and uses biaffine attention [Dozat and Manning, 2017] over the lstm outputs to predict the mentions. The third system takes the outputs from bert [Devlin et al., 2019] and feeds them into a feed-forward neural network to classify candidates into mentions and non mentions. All three systems have the options to output mentions in high recall or high f1 settings; the former is well suited for the coreference task, whereas the latter can be used as a standard mention detector for tasks like nested named entity recognition. We evaluate our models on the conll and the crac data sets for coreference mention detection, and on genia corpora for nested ner.

The contributions of this paper are therefore as follows. First, we show that mention detection performance improved by up to 1.5 percentage points111This performance difference is measured on mention recall, as we follow ?) to use fixed mention/token ratio to compare the mentions selected by their joint system. can be achieved by training the mention detector alone. Second, our best system achieves improvements of 5.3 and 6.2 percentage points when compared with ?)’s neural md system on conll and crac respectively. Third, by using better mentions from our mention detector, we can improve the end-to-end ?) system and the ?) pipeline system by up to 0.7% and 1.7% respectively. Fourth, we show that the state-of-the-art result on nested ner in the genia corpus can be achieved by our best model.

2. Related Work

Mention detection.

Despite neural networks having shown high performance in many natural language processing tasks, the rule-based mention detector of the Stanford deterministic system [Lee et al., 2013] remained frequently used in top performing coreference systems that preceded the development of end-to-end architectures [Clark and Manning, 2015, Clark and Manning, 2016a, Clark and Manning, 2016b], including the best neural network coreference system based on a pipeline architecture, [Clark and Manning, 2016a]. The Stanford Core mention detector uses a set of predefined heuristic rules to select mentions from nps, pronouns and named entity mentions. Many other coreference systems simply use all the nps as the candidate mentions [Björkelund and Kuhn, 2014, Wiseman et al., 2015, Wiseman et al., 2016].

?) first introduced a neural network based end-to-end coreference system in which the neural mention detection part is not separated. This strategy led to a greatly improved performance on the coreference task; however, the mention detection component of their system needs to be trained jointly with the coreference resolution part, hence can not be used separately for applications other than coreference. The Lee et al system has been later extended by ?), ?) and ?). ?) added biaffine attention to the coreference part of the ?) system, improving the system by 0.6%. (Biaffine attention is also used in one of our approaches (Biaffine md), but in a totally different manner, i.e. we use biaffine attention for mention detection while in ?) biaffine attention was used for computing mention-pair scores.) In ?) and ?), the ?) model is substantially improved through the use of elmo [Peters et al., 2018] and bert [Kantor and Globerson, 2019] embeddings.

Other machine learning based mention detectors include ?) and ?). The ?) system takes all the nps as candidates and trains a SVM-based binary classifier to select mentions from all the nps. ?) briefly discuss a neural mention detector that they modified from the ner system of ?). The system uses a bidirectional lstm followed by a FFNN to select mentions from spans up to a maximum width. The system achieved substantial gains on mention F1 when compared with the [Lee et al., 2013] on conll and crac data sets.

Neural Named Entity Recognition. A subtask of mention detection that focuses only on detecting named entity mentions has been studied more frequently. However, most of the proposed approaches treat the ner task as a sequence labelling task, thus cannot be directly applied in tasks that require nested mentions, such as ner in the biomedical domain or coreference. The first neural network based ner model was introduced by ?), who used a cnn to encode the tokens and applied a crf layer on top. After that, many other network architectures for ner md have also been proposed, such as lstm-crf [Lample et al., 2016, Chiu and Nichols, 2016], lstm-crf + elmo [Peters et al., 2018] and bert [Devlin et al., 2019].

Recently, a number of ner systems based on neural network architectures have been introduced to solve nested ner. ?) introduce a stacked lstm-crf approach to solve nested ner in multi-steps. ?) use an exhaustive region classification model. ?) solve the problem in two steps: they first detect the entity head, and then infer the entity boundaries and classes in the second step. ?) infer the nested ner by a sequence-to-sequence model. ?) introduce a boundary aware network to train the boundary detection and the entity classification models in a multi-task learning setting. However, none of those systems can be directly used for coreference, due to the large difference between the settings used in ner and in coreference (e.g. for coreference the mention need to be predicted in a high recall fashion). By contrast, our systems can be easily extended to do nested ner; we demonstrate this by evaluating our system on the genia corpus.

3. System architecture

Mention detection is the task of extracting candidate mentions from the document. For a given document $D$ with $T$ tokens, we define all possible spans in $D$ as $N_{i=1}^{I}$ where $I=\frac{T(T+1)}{2}$, $s_{i},e_{i}$ are the start and the end indices of $N_{i}$ where $1\leq i\leq I$. The task for an md system is to assign all the spans ($N$) a score ($r_{m}$) so that spans can be classified into two classes (mention or non mention), hence is a binary classification problem.

In this paper, we introduce three md systems222The code is available at https://github.com/juntaoy/dali-md that use the most recent neural network architectures. The first approach uses the mention detection part from a start-of-the-art coreference resolution system [Lee et al., 2018], which we refer to as Lee md. We remove the coreference part of the system and change the loss function to sigmoid cross entropy, that is commonly used for binary classification problems. The second approach (Biaffine md) uses a bi-directional lstm to encode the sentences of the document, followed by a biaffine classifier [Dozat and Manning, 2017] to score the candidates. The third approach (Bert md) uses bert [Devlin et al., 2019] to encode the document in the sentence level; in addition, a feed-forward neural network (FFNN) to score the candidate mentions. The three architectures are summarized in Figure 1 and discussed in detail below.

All three architectures are available in two output modes: high f1 and high recall. The high f1 mode is meant for applications that require highest accuracy, such as preprocessing for annotation or nested ner. The high recall mode, on the other hand, predicts as many mentions as possible, which is more appropriate for preprocessing for a coreference system since mentions can be further filtered by the system during coreference resolution. In high f1 mode we output mentions whose probability $p_{m}(i)$ is larger then a threshold $\beta$ such as 0.5. In high recall mode we output mentions based on a fixed mention/word ratio $\lambda$; this is the same method used by ?).

3.1. Lee md

Our first system is based on the mention detection part of the ?) system. The system represents a candidate span with the outputs of a bi-directional lstm. The sentences of a document are encoded bidirectional via the lstms to obtain forward/backward representations for each token in the sentence. The bi-directional lstm takes as input the concatenated embeddings ($(x_{t})_{t=1}^{T}$) of both word and character levels. For word embeddings, GloVe [Pennington et al., 2014] and elmo [Peters et al., 2018] embeddings are used. Character embeddings are learned from convolution neural networks (cnn) during training. The tokens are represented by concatenated outputs from the forward and the backward lstms. The token representations $(x^{*}_{t})_{t=1}^{T}$ are used together with head representations ($h^{*}_{i}$) to represent candidate spans ($N^{*}_{i}$). The $h^{*}_{i}$ of a span is obtained by applying an attention over its token representations ($\{x^{*}_{s_{i}},...,x^{*}_{e_{i}}\}$), where $s_{i}$ and $e_{i}$ are the indices of the start and the end of the span respectively. Formally, we compute $h^{*}_{i}$, $N^{*}_{i}$ as follows:

[TABLE]

[TABLE]

[TABLE]

[TABLE]

where $\phi(i)$ is the span width feature embeddings.

To make the task computationally tractable, the model only considers the spans up to a maximum length of $l$, i.e. $e_{i}-s_{i}<l,\ (s_{i},e_{i})\in N$. The span representations are passed to a FFNN to obtain the raw candidate scores ($r_{m}$). The raw scores are then used to create the probabilities ($p_{m}$) by applying a sigmoid function to the $r_{m}$:

[TABLE]

[TABLE]

For the high recall mode, the top ranked $\lambda T$ spans are selected from $lT$ candidate spans ($\lambda<l$) by ranking the spans in a descending order by their probability ($p_{m}$). For the high f1 mode, the spans that have a probability ($p_{m}$) larger than the threshold $\beta$ are returned.

3.2. Biaffine md

In our second model, the same bi-directional lstm is used to encode the tokens of a document in the sentence level. However, instead of using the concatenations of multiple word/character embeddings, only elmo embeddings are used, as we find in preliminary experiments that the additional GloVe embeddings and character-based embeddings do not improve the accuracy. After obtaining the token representations from the bidirectional lstm, we apply two separate FFNNs to create different representations ($h_{s}/h_{e}$) for the start/end of the spans. Using different representations for the start/end of the spans allows the system to learn important information to identify the start/end of the spans separately. This is an advantage when compared to the model directly using the output states of the lstm, since the tokens that are likely to be the start of the mention and end of the mention are very different. Finally, we employ a biaffine attention [Dozat and Manning, 2017] over the sentence to create a $l_{s}\times l_{s}$ scoring metric ($r_{m}$), where $l_{s}$ is the length of the sentence. More precisely, we compute the raw score for span i ($N_{i}$) by:

[TABLE]

[TABLE]

[TABLE]

where $s_{i}$ and $e_{i}$ are the start and end indices of $N_{i}$, $W_{m}$ is a $d\times d$ metric and $b_{m}$ is a bias term which has a shape of $d\times 1$.

The computed raw score ($r_{m}$) covers all the span combinations in a sentence, to compute the probability scores ($p_{m}$) of the spans we further apply a simple constrain ($s_{i}\leq e_{i}$) such that the system only predict valid mentions. Formally:

[TABLE]

The resulted $p_{m}$ are then used to predict mentions by filtering out the spans according to different requirements (high recall or high f1).

3.3. Bert md

Our third approach is based on the recently introduced bert model [Devlin et al., 2019] in which sentences are encoded using deep bidirectional transformers. Our model uses a pre-trained bert model to encode the documents in the sentence level to create token representations $x_{t}^{*}$. The pre-trained bert model uses WordPiece embeddings [Wu et al., 2016], in which tokens are further split into smaller word pieces as the name suggested. For example in sentence:

We respect ##fully invite you to watch a special edition of Across China .

The token “respectfully” is split into two pieces (“respect” and “fully”). If tokens have multiple representations (word pieces), we use the first representation of the token. An indicator list is created during the data preparation step to link the tokens to the correct word pieces. After obtaining the actual word representations, the model then creates candidate spans by considering spans up to a maximum span length ($l$). The spans are represented by the concatenated representations of the start/end tokens of the spans. This is followed by a FFNN and a sigmoid function to assign each span a probability score:

[TABLE]

[TABLE]

[TABLE]

We use the same methods we used for our first approach (Lee md) to select mentions based on different settings (high recall or high f1) respectively.

3.4. Nested ner

In our evaluation on nested ner we assign each mention pairs $C$ raw scores ($r_{m}$) ($C=1+$ number of ner categories). The first score indicates the likelihood of a span is not a named mentions, the rest of the scores are corresponding to individual ner categories. The probability ($p_{m}$) is then calculated by the softmax function instead of the sigmoid function:

[TABLE]

3.5. Learning

The learning objective of our mention detectors is to learn to distinguish mentions form non-mentions. Hence it is a binary classification problem, we optimise our models on the sigmoid cross entropy.

[TABLE]

where $y_{i}$ is the gold label ($y_{i}\in\{0,1\}$) of ith spans.

For our further experiments on the nested ner we use the softmax cross entropy instead:

[TABLE]

4. Experiments

We ran three series of experiments. The first series of experiments focuses only on the mention detection task, and we evaluate the performance of the proposed mention detectors in isolation. The second series of experiments focuses on the effects of our model on coreference: i.e., we integrate the mentions extracted from our best system into state-of-the-art coreference systems (both end-to-end and the pipeline system). The third series of experiments focuses on the nested ner task. We evaluate our systems both on boundary detection and on the full ner tasks. The rest of this section introduces our experimental settings in detail.

4.1. Data Set

We evaluate our models on two different corpora for both the mention detection and the coreference tasks and one additional corpora for nested ner task, the conll 2012 English corpora [Pradhan et al., 2012], the crac 2018 corpora [Poesio et al., 2018] and the genia [Kim et al., 2003] corpora.

The conll data set is the standard reference corpora for coreference resolution. The English subset consists of 2802, 342, and 348 documents for the train, development and test sets respectively. The conll data set is not however ideal for mention detection, since not all mentions are annotated, but only mentions involved in coreference chains of length $>1$. This has a negative impact on learning since singleton mentions will always receive negative labels.

The crac corpus uses data from the arrau corpus [Uryupina et al., 2019]. arrau consists of texts from four very distinct domains: news (the rst subcorpus), dialogue (the trains subcorpus) and fiction (the pear stories). This corpus is more appropriate for studying mention detection as all mentions are annotated. As done in the crac shared task, we used the rst portion of the corpora, consisting of news texts (1/3 of the penn Treebank). Since none of the state-of-the-art coreference systems predict singleton mentions, a version of the crac dataset with singleton mentions excluded was created for the coreference task evaluation.

The genia corpora is one of the main resources for studying nested ner. We use the genia v3.0.2 corpus and preprocess the dataset following the same settings of ?) and ?). Historically, the dataset has been split into two different ways: the first approach splits the data into two sets (train and test) by 90:10 (genia90), whereas the second approach further creates a development set by splitting the data into 81:9:10 (genia81). We evaluate our model on both approaches to make the fair comparisons with previous work. For evaluation on genia90, since we do not have a development set, we train our model for 40K steps (20 epochs) and take evaluate on the final model.

4.2. Evaluation Metrics

For our experiments on the mention detection or nested ner, we report recall, precision and F1 scores for mentions. For our evaluation that involves the coreference system, we use the official conll 2012 scoring script to score our predictions. Following standard practice, we report recall, precision, and F1 scores for MUC, B3 and CEAF${}_{\phi_{4}}$ and the average F1 score of those three metrics.

4.3. Baseline System

For the mention detection evaluation we use the ?) system as baseline. The baseline is trained end-to-end on the coreference task and we use as baseline the mentions predicted by the system before carrying out coreference resolution.

For the coreference evaluation we use the top performing ?) system as our baseline for the end-to-end system, and the ?) system as our baseline for the pipeline system. During the evaluation, we slightly modified the ?) system to allow the system to take the mentions predicted by our model instead of its internal mention detector. Other than that we keep the system unchanged.

For the nested ner we compare our system with ?) and ?) on genia81 and with ?), ?) an ?) on genia90.

4.4. Hyperparameters

For our first model (Lee md) we use the default settings of ?). For word embeddings the system uses 300-dimensional GloVe embeddings [Pennington et al., 2014] and 1024-dimensional elmo embeddings [Peters et al., 2018]. The character-based embeddings are produced by a convolution neural network (cnn) which has a window sizes of 3, 4, and 5 characters (each has 50 filters). The characters embeddings (8-dimensional) are randomly initialised and learned during the training. The maximum span width is set to 30 tokens.

For our Biaffine md model, we use the same lstm settings and the hidden size of the FFNN as our first approach. For word embeddings, we only use the elmo embeddings [Peters et al., 2018].

For our third model (Bert md), we fine-tune on the pre-trained bert${}_{\textsc{base}}$ that consists of 12 layers of transformers. The transformers use 768-dimensional hidden states and 12 self-attention heads. The WordPiece embeddings [Wu et al., 2016] have a vocabulary of 30,000 tokens. We use the same maximum span width as in our first approach (30 tokens).

The detailed network settings can be found in Table 1.

5. Results and Discussions

In this section, we first evaluate the proposed models in isolation on the mention detection task. We then integrate the mentions predicted by our system into coreference resolution systems to evaluate the effects of our md systems on the downstream applications. Finally we evaluate our system on the nested ner task.

5.1. Mention Detection Task

Evaluation on the conll data set. For mention detection on the conll data set, we first take the best model from ?) and use its default mention/token ratio ($\lambda=0.4$) to output predicted mentions before coreference resolution. We use this as our baseline for the high recall setting. We then evaluate all three proposed models with the same $\lambda$ as that of the baseline. As a result, the number of mentions predicted by different systems is the same, which means mention precision will be similar. Thus, for the high recall setting we compare the systems by mention recall. As we can see from Table 2, the baseline system already achieved a reasonably good recall of 96.6%. But even when compared with such a strong baseline, by simply separately training the mention detection part of the baseline system, the stand-alone Lee md achieved an improvement of 0.7 p.p. This indicates that mention detection task does not benefit from joint mention detection and coreference resolution. The Bert md achieved the same recall as the Lee md, but Bert md uses a much deeper network and is more expensive to train. By contrast, the Biaffine md uses the simplest network architecture among the three approaches, yet achieved the best results, outperforming the baseline by 0.9 p.p. (26.5% error reduction).

**Evaluation on the crac data set333As the ?) system does not predict singleton mentions, the results on crac data set in Table 2 are evaluated without singleton mentions, whereas the results reported in Table 3 are evaluated with singleton mentions included. ** For the crac data set, we train the ?) system end-to-end on the reduced corpus with singleton mentions removed and extract mentions from the system by set $\lambda=0.4$. We then train our models with the same $\lambda$ but on the full corpus, since our mention detectors naturally support both mention types (singleton and non-singleton mentions). Again, the baseline system has a decent recall of 95.4%. Benefiting from the singletons, our Lee md and Biaffine md models achieved larger improvements when compared with the gains achieved on the conll data set. The largest improvement (1.8 p.p.) is achieved by our Biaffine md model with an error reduction rate of 39.1%. Bert md achieved a relatively smaller gain (0.8 p.p.) when compared with the other models; this might be a result of the difference in corpus size between crac and conll data set. (The crac corpus is smaller than the conll data set.)

Model complexity and speed To give an idea of the difference in model complexity and inference speed, we listed the number of trainable parameters and the inference speed of our models on conll and crac in Table 4. The Biaffine md model consists of the lowest number of trainable parameters among all three models, it uses 85% or 3% parameters when compared with the Lee md and Bert md respectively. In addition, the Biaffine md is also the fastest model on both conll and crac datasets, which is able to process 8.3 conll or 6.7 crac documents per second444All the speed test is conducted on a single GTX 1080ti GPU..

Comparison with the State-of-the-art. We compare our best system Biaffine md with the rule-based mention detector of the Stanford deterministic system [Lee et al., 2013] and the neural mention detector of ?). For high f1 setting we use the common threshold ($\beta=0.5$) for binary classification problems without tuning. For evaluation on conll we create in addition a variant of the high recall setting (balance) by setting $\lambda=0.2$; this is because we noticed that the score differences between the high recall and high f1 settings are relatively large (see Table 3). The score differences between our two settings on crac data set are smaller; this might because the crac data set annotated both singleton and non-singleton mentions, hence the models are trained in a more balanced way. Overall, when compared with the best-reported system [Poesio et al., 2018], our high f1 settings outperforms their system by large margin of 5.3% and 6.2% on conll and crac data sets respectively.

5.2. Coreference Resolution Task

We then integrate the mentions predicted by our best system into the coreference resolution system to evaluate the effects of our better mention detectors on the downstream application.

Evaluation with an end-to-end system. We first evaluate our Biaffine md in combination with the end-to-end ?) system. We slightly modified the system to feed the system mentions predicted by our mention detector. As a result, the original mention selection function is switched off, we keep all the other settings (include the mention scoring function) unchanged. We then train the modified system to obtain a new model. As illustrated in Table 5, the model trained using mentions supplied by our Biaffine md achieved a F1 score slightly lower than the original end-to-end system, even though our mention detector has a better performance.

We think the performance drop might be the result of two factors. First, by replacing the original mention selection function, the system actually becomes a pipeline system, thus cannot benefit from joint learning. Second, the performance difference between our mention detector and the original mention selection function might not be large enough to deliver improvements on the final coreference results. To test our hypotheses, we evaluated our Biaffine md with two additional experiments.

In the first experiment, we enabled the original mention selection function and fed the system slightly more mentions. More precisely, we configured our Biaffine md to output 0.5 mention per token instead of 0.4 i.e. $\lambda=0.5$. As a result, the coreference system has the freedom to select its own mentions from a candidate pool supplied by our Biaffine md. After training the system with the new setting, we get an average F1 of 72.6% (see table 5), which narrows the performance gap between the end-to-end system and the model trained without the joint learning. This confirms our first hypothesis that by downgrading the system to a pipeline setting does harm the overall performance of the coreference resolution.

For our second experiment, we used the ?) instead. The ?) system is an extended version of the ?) system, hence they share most of the network architecture. The ?) has a lower performance on mention detection (93.5% recall when $\lambda=0.4$), which creates a large (4%) difference when compared with the recall of our Biaffine md. We train the system without the joint learning, and the newly trained model achieved an average F1 of 67.7% and this is 0.5 better than the original end-to-end ?) system (see table 5). This confirms our second hypothesis that a larger gain on mention recall is needed in order to show improvement on the overall system.

We further evaluated the ?) system on the crac data set. We first train the original ?) on the reduced version (with singletons removed) of the crac data set to create a baseline. As we can see from Table 5, the baseline system has an average F1 score of 68.4%. We then evaluate the system with mentions predicted by our Biaffine md, we experiment with both joint learning disabled and enabled. As shown in Table 5, the model without joint learning achieved an overall score 0.1% lower than the baseline, but the new model has clearly a better recall on all three metrics when compared with the baseline. The model trained with joint learning enabled achieved an average F1 of 69.1% which is 0.7% better than the baseline.

Evaluation with a pipeline system. We then evaluated our best model (Biaffine md) with a pipeline system. We use the best-reported pipeline system by ?) as our baseline. The original system used the rule-based mention detector from the Stanford deterministic coreference system [Lee et al., 2013] (a performance comparison between the ?) and our Biaffine md can be found in Table 3). We modified the preprocessing pipeline of the system to use mentions predicted by our Biaffine md. We ran the system with both mentions from the high recall and balance settings, as both settings have reasonable good mention recall which is required to train a coreference system. After training the system with mentions from our Biaffine md, the newly obtained models achieved large improvements of 0.8% and 1.7% for high recall and balance settings respectively. This suggests that the ?) system works better on a smaller number of high-quality mentions than a larger number but lower quality mentions. We also noticed that the speed of the ?) system is sensitive to the size of the predicted mentions, both training and testing finished much faster when tested on the balance setting. We did not test the ?) system on the crac data set, as a lot of effects are needed to fulfil the requirements of the preprocessing pipeline, e.g. predicted parse trees, named entity tags. Overall our Biaffine md showed its merit on enhancing the pipeline system.

Summary In summary, our results suggest that the picture regarding using our Biaffine md for coreference resolution is more mixed than with coreference annotation as discussed in the previous Section (and with nested ner as shown in the following Section). Our model clearly improves the results of best current pipeline system; when used with a top performing end-to-end system, it improves the performance with the crac dataset but not with conll.

5.3. Nested Named Entity Recognition Task

We then extend our best system (Biaffine md) to do nested ner task.

Evaluation on genia81 We first evaluate our system on the split (genia81) with the development set. We first run our Biaffine md on boundary detection task which do not require any modification on the system. On boundary detection our system achieved 80.0%, 82.3% and 81.1% for recall precision and F1 score respectively. Our results out perform the previous state-of-the-art system [Zheng et al., 2019] by 2.8% (F1 score).

We then extend our system to predict full ner task, Table 6 and Table 7 show our overall and individual category results on the genia81 test set respectively. As we can see from Table 6 our system outperforms the previous state-of-the-art system by 1.1 percentage points. In addition, our system also achieved a much better results on three out of five categories (see Table 7). Overall our system achieved the new state-of-the-art on the genia81 data for both boundary detection and full nested ner tasks.

Evaluation on genia90 Next, we evaluate our system on the other split of the corpora (genia90), in this setting we have a larger training set but do not have a development set. After we train our model for 20 epochs, our final model outperforms the previous state-of-the-art by 0.3% (see Table 8). In Table 9 we present our detailed scores for individual categories, since both ?) and ?) did not report the detailed scores, we compare our system with ?). Our system outperforms theirs on all five categories.

6. Conclusions

In this work, we compare three neural network based approaches for mention detection. The first model is a modified version of the mention detection part of a top performing coreference resolution system [Lee et al., 2018]. The second model used elmo embeddings together with a bidirectional lstm, and with a biaffine classifier on top. The third model adapted the bert model that based on the deep transformers and followed by a FFNN. We assessed the performance of our models in mention detection, coreference and nested ner tasks. In the evaluation of mention detection, our proposed models reduced up to 26% and 39% of the recall error when compared with the strong baseline on conll and crac data sets in a high recall setting. The same model (Biaffine md) outperforms the best performing system on the conll and crac by large 5-6% in a high f1 setting. In term of the evaluation on coreference resolution task, by integrating our mention detector with the state-of-the-art coreference systems, we improved the end-to-end and pipeline systems by up to 0.7% and 1.7% respectively. The evaluation on the nested ner task showed that despite our model is not specifically designed for the task, we achieved the state-of-the-art on the genia corpora. Overall, we introduced three neural mention detectors and showed that the improvements achieved on the mention detection task can be transferred to the downstream coreference resolution task. In addition, our model is robust enough be used for nested ner task.

7. Acknowledgments

This research was supported in part by the DALI project, ERC Grant 695662.

8. Bibliographical References