Causality Extraction based on Self-Attentive BiLSTM-CRF with Transferred Embeddings

TL;DR

This paper introduces SCITE, a neural causality extractor using self-attention and transferred embeddings, formulated as a sequence labeling task, which outperforms existing methods without extensive feature engineering.

Contribution

The paper proposes a novel causality extraction model based on BiLSTM-CRF with transferred Flair embeddings and self-attention, reducing reliance on domain knowledge and manual feature engineering.

Findings

Significant improvement over baseline methods in causality extraction accuracy

Effective use of transferred embeddings to address data scarcity

Enhanced dependency learning through multihead self-attention

Abstract

Causality extraction from natural language texts is a challenging open problem in artificial intelligence. Existing methods utilize patterns, constraints, and machine learning techniques to extract causality, heavily depending on domain knowledge and requiring considerable human effort and time for feature engineering. In this paper, we formulate causality extraction as a sequence labeling problem based on a novel causality tagging scheme. On this basis, we propose a neural causality extractor with the BiLSTM-CRF model as the backbone, named SCITE (Self-attentive BiLSTM-CRF wIth Transferred Embeddings), which can directly extract cause and effect without extracting candidate causal pairs and identifying their relations separately. To address the problem of data insufficiency, we transfer contextual string embeddings, also known as Flair embeddings, which are trained on a large corpus in our task. In addition, to improve the performance of causality extraction, we introduce a multihead self-attention mechanism into SCITE to learn the dependencies between causal words. We evaluate our method on a public dataset, and experimental results demonstrate that our method achieves significant and consistent improvement compared to baselines.

Tables

Table 1. Table 1: The intermediate result of running tag2triplet when we input the sentence S 𝑆 S and its corresponding tag sequence S t ​ a ​ g subscript 𝑆 𝑡 𝑎 𝑔 S_{tag} , and we highlight the correct combination of candidates in bold.

$S$	… $[$the chronic inflammation$]{}_{E_0}$ … $[$Helicobacter pylori infection$]{}_{E_1}$ … $[$an increased acid production$]{}_{E_2}$ …
$S_{tag}$	$[$B-Emb I-Emb I-Emb$]{}_{E_0}$	$[$B-C I-C I-C $]{}_{E_1}$	$[$B-E I-E I-E I-E$]{}_{E_2}$
Index	${[5,6,7]}_{E_0}$	${[17,18,19]}_{E_1}$	${[22,23,24,25]}_{E_2}$
Out-degree	$1_{E_0}$	$1_{E_1}$	$0_{E_2}$
In-degree	$1_{E_0}$	$0_{E_1}$	$1_{E_2}$
Candidates	($E_0,E_2$)	($E_1,E_0$)	($E_1,E_2$)
Combinations	($E_0,E_2$), ($E_1,E_0$)	($E_0,E_2$), ($E_1,E_2$)	($E_1,E_0$), ($E_1,E_2$)
Out-degree	($1_{E_0}$, $1_{E_1}$, $0_{E_2}$)	($1_{E_0}$, $1_{E_1}$, $0_{E_2}$)	($1_{E_0}$, $0_{E_1}$, $0_{E_2}$)
In-degree	($1_{E_0}$, $0_{E_1}$, $1_{E_2}$)	($0_{E_0}$, $0_{E_1}$, $1_{E_2}$)	($1_{E_0}$, $0_{E_1}$, $1_{E_2}$)

Table 2. Table 2: Statistics of different types of causal tags for the dataset

Tag Type	Training Set	Test Set
B-C	1308	236
I-C	1421	229
B-E	1268	238
I-E	1230	230
B-Emb	55	9
I-Emb	55	16
Sum	5337	958

Table 3. Table 3: Comparison in precision (P), recall (R), and F1-score (F) on the test set with baselines.

Model	P	R	F
CausalNet	0.6211	0.5372	0.5761
Rules-Bayesian	0.6042	0.5878	0.5959
IDCNN-softmax	0.7455$\pm$0.0142	0.7074$\pm$0.0168	0.7258$\pm$0.0105
IDCNN-CRF	0.7442$\pm$0.0225	0.7142$\pm$0.0122	0.7288$\pm$0.0160
BiLSTM-softmax	0.7744$\pm$0.0183	0.7622$\pm$0.0114	0.7682$\pm$0.0138
CLSTM-BiLSTM-CRF	0.8144$\pm$0.0284	0.7412$\pm$0.0073	0.7757$\pm$0.0107
CCNN-BiLSTM-CRF	0.8069$\pm$0.0199	0.7520$\pm$0.0227	0.7780$\pm$0.0075
BiLSTM-CRF	0.7837$\pm$0.0061	0.7932$\pm$0.0087	0.7884$\pm$0.0072
BERT-BiLSTM-CRF	0.8277$\pm$0.0058	0.8209$\pm$0.0093	0.8243$\pm$0.0049
Flair+CLSTM-BiLSTM-CRF	0.8403$\pm$0.0090	0.8284$\pm$0.0125	0.8343$\pm$0.0106
ELMo-BiLSTM-CRF	0.8361$\pm$0.0135	0.8399$\pm$0.0063	0.8379$\pm$0.0092
Flair-BiLSTM-CRF	0.8414$\pm$0.0079	0.8351$\pm$0.0141	0.8382$\pm$0.0092
SCITE (Flair+CCNN-BiLSTM-MHSA-CRF)	0.8333$\pm$0.0042	0.8581$\pm$0.0021	0.8455$\pm$0.0028
SCITE (based on general tagging scheme)	0.7609$\pm$0.0170	0.7757$\pm$0.0136	0.7682$\pm$0.0145

Table 4. Table 4: Comparison of predicted tags concerning “C” (cause), “E” (effect) and “Emb” (embedded causality) in precision (P), recall (R), and F1-score (F) on the test set.

Model	C-P	C-R	C-F	E-P	E-R	E-F	Emb-P	Emb-R	Emb-F
BiLSTM-CRF	0.8810	0.8628	0.8718	0.8928	0.8897	0.8913	0.4343	0.0960	0.1567
Flair-BiLSTM-CRF	0.8995	0.8843	0.8917	0.9294	0.8885	0.9084	0.8556	0.1360	0.2197
SCITE	0.8999	0.8998	0.8998	0.9272	0.9021	0.9144	0.8489	0.1920	0.2947

Table 5. Table 5: Ablation analysis of our proposed model SCITE. “All” denotes the complete SCITE model, i.e., the Flair+CCNN-BiLSTM-MHSA-CRF model, while “-” denotes removing the component from the SCITE.

Model	Setting	F
SCITE	All	0.8455
Flair-BiLSTM-MHSA-CRF	-CCNN	0.8438
Flair-BiLSTM-CRF	-CCNN -MHSA	0.8382
BiLSTM-MHSA-CRF	-Flair -CCNN	0.8137
BiLSTM-CRF	-Flair -CCNN -MHSA	0.7884

Table 6. Table 6: Results of causality extraction, where “C” represents “cause”, and “E” represents “effect”. We italicize correct results and highlight the incorrect results in bold.

Sentence 1	[The damages] caused by [mudslides], [tremors], [subsidence], [superficial or underground water] were verified, as well as [swelling clay soils].
True Triplets	{[mudslides], C-E, [The damages]}, {[tremors], C-E, [The damages]}, {[subsidence], C-E, [The damages]}, {[superficial or underground water], C-E, [The damages]}, {[swelling clay soils], C-E, [The damages]}
SCITE	{[mudslides], C-E, [The damages]}, {[tremors], C-E, [The damages]}, {[subsidence], C-E, [The damages]}, {[superficial], C-E, [The damages]}, {[underground water], C-E, [The damages]}
Flair-BiLSTM-CRF	{[mudslides], C-E, [The damages]}, {[tremors], C-E, [The damages]}, {[subsidence], C-E, [The damages]}, {[underground water], C-E, [The damages]} None
BiLSTM-CRF	{[mudslides], C-E, [The damages]}, {[tremors], C-E, [The damages]}, {[subsidence], C-E, [The damages]}, {[underground water], C-E, [The damages]} None
Sentence 2	This year’s Nobel Laureates in Physiology or Medicine made the remarkable and unexpected discovery that [inflammation] in the stomach as well as [ulceration] of the stomach or duodenum is the result of [an infection] of the stomach caused by [the bacterium Helicobacter pylori].
True Triplets	{[an infection], C-E, [inflammation]}, {[an infection], C-E, [ulceration]}, {[the bacterium Helicobacter pylori], C-E, [an infection]}
SCITE	{[an infection], C-E, [inflammation]}, {[an infection], C-E, [ulceration]}, {[the bacterium Helicobacter pylori], C-E, [an infection]}
Flair-BiLSTM-CRF	{[the bacterium Helicobacter pylori], C-E, [inflammation]}, {[the bacterium Helicobacter pylori], C-E, [ulceration]}, {[the bacterium Helicobacter pylori], C-E, [an infection]}
BiLSTM-CRF	{[the bacterium Helicobacter pylori], C-E, [the remarkable and unexpected discovery]}, None, {[the bacterium Helicobacter pylori], C-E, [an infection]}

Equations

$$c_i={\mathbf{w}}^{\mathrm{T}}{\mathbf{r}}_{i:i+l-1}+b,$$

$$\hat{\mathbf{c}}=[c_1,c_2,...,c_{s-l+1}],$$

$${\mathbf{c}}_t=[\widetilde{c_1},\widetilde{c_2},...,\widetilde{c_f}]$$

$${\mathbf{f}}_t^{CharLM}=[\overrightarrow{{\mathbf{h}}_{end+1}^t},\overleftarrow{{\mathbf{h}}_{start-1}^t}]$$

$${\mathbf{i}}_t$$

$${\mathbf{f}}_t$$

$${\mathbf{o}}_t$$

$$\widetilde{{\mathbf{m}}_t}$$

$${\mathbf{m}}_t$$

$${\mathbf{h}}_t$$

$${\mathbf{h}}_{\mathbf{t}}=[\overrightarrow{{\mathbf{h}}_t},\overleftarrow{{\mathbf{h}}_t}]$$

$$\mathbf{M}=MultiHead(\mathbf{H},\mathbf{H},\mathbf{H})=Concat({\mathbf{head}}_1...,{\mathbf{head}}_h)$$

$$where{\mathbf{head}}_i=Attention(\mathbf{H}{\mathbf{W}}_i^Q,\mathbf{H}{\mathbf{W}}_i^K,\mathbf{H}{\mathbf{W}}_i^V)$$

$$Attention(\mathbf{H}{\mathbf{W}}_i^Q,\mathbf{H}{\mathbf{W}}_i^K,\mathbf{H}{\mathbf{W}}_i^V)=softmax(\frac{(\mathbf{H}{\mathbf{W}}_i^Q)(\mathbf{H}{\mathbf{W}}_i^K{)}^{\mathrm{T}}}{\sqrt{d}})(\mathbf{H}{\mathbf{W}}_i^V)$$

$$score\left(S,y\right)=\sum _{i=0}^n{\mathbf{A}}_{y_i,y_{i+1}}+\sum _{i=1}^n{\mathbf{P}}_{i,y_i},$$

$$p\left(y\mid S\right)=\frac{e^{score\left(S,y\right)}}{{\sum }_{\tilde{y}\in Y_S}{e}^{score\left(S,\tilde{y}\right)}},$$

$$\log \left(p\left(y\mid S\right)\right)=score\left(S,y\right)-\log \left(\sum _{\tilde{y}\in Y_S}{e}^{score\left(S,\tilde{y}\right)}\right),$$

$$y^{\ast }=arg\underset{\tilde{y}\in Y_S}{\max }score\left(S,\tilde{y}\right)$$

$$P$$

$$R$$

$$F$$

Full text

[TABLE]

1 Introduction

Natural language text contains considerably causal knowledge, as shown in Fig. 1. In recent years, causality extraction has become increasingly important for many natural language processing tasks, such as information retrieval [1, 2], event prediction [3, 4], question answering [5, 6, 7], generating future scenarios [8, 9], decision processing [10], medical text mining [11, 12, 13] and behavior prediction [14]. However, due to the ambiguity and diversity of natural language texts, causality extraction remains a hard NLP problem to solve.

Traditional methods for causality extraction can be divided into two categories: methods based on patterns [1, 11, 15, 16] (Section 5.1), and methods based on a combination of patterns and machine learning techniques [5, 17, 18, 19] (Section 5.2). The former often has poor cross-domain applicability, fails to balance precision and recall and may require extensive domain knowledge to solve problems in a particular area. The latter usually requires considerable human effort and time on feature engineering, relying heavily on the manual selection of textual features. Generally, it divides causality extraction into two subtasks, candidate causal pairs extraction and relation classification (filtering noncausal pairs). The results of candidate causal pairs extraction may affect the performance of relation classification and generate cascading errors.

Zheng et al. [20] first proposed a tagging scheme that makes it possible for models to extract entities and relations simultaneously. In their tagging scheme, they apply a cartesian product of the entity mention tags and the relation type tags, and then assign a unique tag that encodes entity mentions and relation types for each word. Inspired by their novel idea, we focus on a causal triplet that is composed of two event entities and their relation. For instance, the sentence in Fig. 1 contains a causal triplet: “{financial stress, cause-effect, divorce}”. Thus, we can model the causal triplets directly, rather than breaking causality extraction into two subtasks. Based on the motivations, we formulate causality extraction into a sequence tagging problem and propose a causality tagging scheme (Section 2.1) to achieve direct causality extraction. However, the tagging scheme proposed by Zheng et al. [20] cannot identify the overlapping relations in a sentence; it only considers situations where an entity belongs to one triplet: if an entity participates in multiple relations, its tag should not be unique. To address this problem, we design a tag2triplet algorithm (Section 2.2) to handle multiple causal triplets and embedded causal triplets in the same sentence. Finally, we combine the causality tagging scheme with a deep learning architecture (Section 2.3) to minimize feature engineering while efficiently modeling causal relations in natural language text.

We notice that some researchers have also proposed deep learning technique-based methods for causality extraction in recent years (Section 5.3). Although their works are commendable, some works [21, 22, 23, 24] are only a classification of causal relations rather than an extraction of complete causal triplets, and others [25, 26] mainly focus on the identification of the linguistic expressions for causality instead of the commonsense causality extraction in this paper.

By applying our causality tagging scheme, we use the model based on BiLSTM-CRF [27] to extract causal triplets directly. However, we find that two obstacles hinder the further improvement of the performance of the deep learning model.

First, it is difficult to train a superior deep learning model without any prior knowledge in the case of data insufficiency in the existing corpus [28, 29, 30]. To alleviate this problem, we incorporate Flair embeddings [31] into our task, which use the internal states of a character language model trained on a large corpus to create word embeddings (Section 2.3.2). Experimental results show that this contextual string embedding that has initiated a new technology trend in NLP can drastically improve the performance of causality extraction.

Second, in terms of their positions in the text, cause and effect are sometimes far from each other, as Fig. 2 shows. The long-range dependency in the causal triplet creates difficulty and ambiguity in the deep learning model, but a set of logical rules based on dependency trees can easily and accurately extract such triplets. To learn this kind of long-range dependency between cause and effect, we introduce the multihead self-attention mechanism [32] into our model (Section 2.3.4). Unlike the LSTM-based model that recursively processes each word, the self-attention mechanism can conduct direct connections between two arbitrary words in a sentence and thus allows unimpeded information flow through the network [33].

The contributions of this paper can be summarized as follows:

We design a novel causality tagging scheme to directly extract causalities in texts and can easily transform the causality extraction into a sequence labeling task and handle multiple causal triplets and embedded causal triplets in the same sentence. 2. 2.

Based on our causality tagging scheme, we propose SCITE (Self-attentive BiLSTM-CRF wIth Transferred Embeddings), a neural-based causality extractor with transferred contextual string embeddings trained on a large corpus. To the best of our knowledge, we are the first to transfer Flair embeddings into causality extraction. 3. 3.

We introduce the multihead self-attention mechanism into SCITE, which enables the model to capture long-range dependencies between cause and effect. 4. 4.

Extensive experimental results (Section 3) and further analysis (Section 4) show that our method achieves significant and consistent improvement compared to other baselines. We release the code and dataset to the research community for further research 111https://github.com/Das-Boot/scite.

2 Method

2.1 Causality Tagging Scheme

We use the “BIO” (begin, inside, other) and “C, E, Emb” (cause, effect, embedded causality) signs to represent the position information of the words and the semantic roles of the causal events, respectively, where embedded causality [30] indicates that a causal event has different roles of causality in different triplets. Fig. 3 is an example of an embedded causality in a sentence. The example sentence contains two causal triplets: “{the chronic inflammation, cause-effect, an increased acid production}” and “{Helicobacter, cause-effect, the chronic inflammation}”, note that “the chronic inflammation” is the cause in the first triplet and the effect in the second triplet.

Fig. 4 shows an example of such causality sequence tagging. Based on our causality tagging scheme, we label the causal event entities “chronic inflammation”, “Helicobacter pylori infection” and “increased acid production” separately with our special tags. Specifically, tag “O” represents the“other”, which means that the corresponding word is irrelevant in any causality components. Tag “B-C” represents the “cause begin”, tag “I-C” represents the “cause inside”, tag “B-E” represents the “effect begin”, tag “I-E” represents the “effect inside”, tag “B-Emb” represents the “embedded causality begin”, and tag “I-Emb” represents the “embedded causality inside”. Thus, the total number of tags is $N_{t}=7$.

2.2 From Tag Sequence to Causal Triplets

We design a tag2triplet algorithm for automatically obtaining the final extracted triplets from the tag sequence in Fig. 4. To better illustrate this algorithm, we define two types of causality: simple causality and complex causality.

2.2.1 The Case of Simple Causality

Simple causality can be classified into two types:

There is only one cause or one effect in the sentence, and there is no embedded causality, that is, $N_{C}=1$ or $N_{E}=1$ and $N_{Emb}=0$, where $N_{C}$, $N_{E}$, and $N_{Emb}$ respectively indicate the number of tags “B-C”, “B-E”, and “B-Emb” in the sentence. The example sentences in Fig. 1 and Fig. 2 are both of this type of causality. 2. 2.

There are multiple causes and effects in the sentence, and there is no embedded causality, i.e., $N_{C}>1$, $N_{E}>1$ and $N_{Emb}=0$. In addition, for each causal triplet in the sentence, there must be at least one causal triplet that shares the same cause or effect. The example sentence in Fig. 5 is this type of causality.

2.2.2 The Case of Complex Causality

Complex causality has the following two types:

There is embedded causality in the sentence, that is, $N_{C}>0$, $N_{E}>0$ and $N_{Emb}>0$. The example sentence in Fig. 3 is this type of causality. 2. 2.

There are multiple causes and effects in the sentence, and there is no embedded causality, i.e., $N_{C}>1$, $N_{E}>1$ and $N_{Emb}=0$. In addition, in all the causal triplets in the sentence, there must be at least one causal triplet that does not share the same cause or effect with any other triplets. The example sentence in Fig. 6 is this type of causality. Note that the distribution of causality in the sentence of Fig. 5 is different from that in Fig. 6: each causal triplet in the former is mixed together, and each causal triplet in the latter is separated.

2.2.3 Tag2triplet Algorithm

The tag2triplet algorithm is described in Algorithm 1. We elaborate on the tag2triplet algorithm by taking the sentence $S$ and its corresponding tag sequence $S_{tag}$ as an example in Fig. 4; the intermediate result is shown in Table 1.

First, we count the out-degree and in-degree for causality and find the index of causality in $S_{tag}$. Specifically, the out-degree of “cause” is recorded as 1, the in-degree of “effect” is recorded as 1, and the out-degree and in-degree of “embedded causality” are both recorded as 1. Then, we determine whether the $S$ is simple causality or complex causality according to the number and the distribution of each causal tag: “C”, “E” and “Emb”. In $S_{tag}$, $N_{Emb}=1$, and thus, $S$ is complex causality. Then, we apply a Cartesian product of the causal entities composed of causal tags to generate the candidates of the causal triplet. In Table 1, the candidate “($E_{0},E_{2}$)” represents the causal triplet “{the chronic inflammation, cause-effect, an increased acid production}.

Next, combination returns $i$ length subsequences of triplets from the input candidates. After that, we determine whether the out-degree and in-degree of each combination of candidates are consistent with the original out-degree and in-degree of $S_{tag}$. Then, we determine whether the combination matches the rules according to the coordinating conjunction in $S$, for example, if there is a coordinating conjunction “and” between adjacent causes in the same clause, then the two causes will form their respective causal triplets with the same effect, as in the “bacteria” and “comedonal debris” in Fig. 5. Finally, we select the combination with the shortest distance from the combinations that passed the checks as the extracted causal triplets. Since only one combination “($E_{0},E_{2}$), ($E_{1},E_{0}$)” passed all the checks, we directly output it as the final result.

2.3 SCITE

Fig. 7 gives the main structure of our model SCITE for causality sequence labeling. We take the input sentence $S=\{x_{t}\}^{n}_{t=1}$ and its corresponding label sequence $y=\{y_{i}\}^{n}_{t=1}$ as an example to introduce each component of SCITE from bottom to top as follows, where $n$ is the length of the $S$.

2.3.1 CNN for Character Representations

To capture task-specific subword features, we take the same convolutional neural network [34] (CNN) architecture as Ma and Hovy [35], using a one-layer CNN structure followed by a max-over-time pooling operation [36] to learn character-level representations. The process is depicted on the left side of Fig. 7.

Specifically, let $\bm{r}_{i}\in\mathbb{R}^{m}$ be the $m$-dimensional character vector corresponding to the $i$-th character in the word $x_{t}$ (the length of $x_{t}$ is $s$). A convolution operation involves a filter $\bm{w}\in\mathbb{R}^{lm}$, which is applied to a window of $l$ characters to produce a new feature. For example, a feature $c_{i}$ is generated from a window of characters $\bm{r}_{i:i+l-1}$ 222In general, let $\bm{r}_{i:i+j}$ refer to the concatenation of character vectors $\bm{r}_{i}$, $\bm{r}_{i+1}$, …, $\bm{r}_{i+j}$ by

[TABLE]

where $b$ is a bias term. This filter is applied to each possible window of character vectors in the word $\{\bm{r}_{1:l},\bm{r}_{2:h+1},...,\bm{r}_{s-l+1:s}\}$ to produce a feature map

[TABLE]

with $\bm{\hat{c}}\in\mathbb{R}^{s-l+1}$. Then, we take the maximum value $\tilde{c}=max\{\bm{\hat{c}}\}$ as the feature corresponding to this particular filter. Thus, denoting that the number of filters is $f$, the character representation $\bm{c}_{t}$ for word $x_{t}$ is given as:

[TABLE]

2.3.2 Transferring Contextualized Representations Learned from Large Corpus

In recent years, deep learning has ushered in incredible advances in natural language processing (NLP) tasks due to its powerful representation learning ability. However, in the case of data insufficiency of the existing corpus, the data-hungry nature of deep learning limits the performance of our neural-based model in causality extraction. The recent development of contextualized language representation models [37, 38, 31] trained on large corpora shed light on the possibility of transfer learning.

In this paper, we use transfer learning to alleviate the problem of data insufficiency. Specifically, we propose to transfer the Flair embeddings [31], which were derived from a character-level language model (CharLM) trained on a 1-billion word benchmark corpus [39] to our task. This CharLM consists of a forward language model (fLM) and a backward language model (bLM). Following Akbik et al. [31], we extract the output hidden state $\overrightarrow{\bm{h}^{t}_{end+1}}$ from the fLM after the last character $r^{t}_{end}$ of the word $x_{t}$. Similarly, we obtain the output hidden state $\overleftarrow{\bm{h}^{t}_{start-1}}$ from the bLM before the first character $r^{t}_{start}$ of the word $x_{t}$. Then, both output hidden states are concatenated to form the final embedding $\bm{f}^{CharLM}_{t}$ of the word $x_{t}$ as follows:

[TABLE]

Finally, we concatenate transferred Flair embeddings $\bm{f}^{CharLM}_{t}$ and the character representations $\bm{c}_{t}$ with the word embeddings $\bm{e}_{t}$ pretrained by Komninos and Manandhar [40] and feed them into a BiLSTM layer.

2.3.3 BiLSTM

Long short-term memory (LSTM) [41] is a particular recurrent neural network (RNN) that overcomes the vanishing and exploding gradient problems [42] of traditional RNN models. Through the specifically designed gate structure of LSTM, the model can selectively save context information. The basic unit of the LSTM architecture is a memory block, which includes a memory cell (denoted as $\bm{m}$) and three adaptive multiplication gates (i.e., an input gate $\bm{i}$, a forget gate $\bm{f}$ and an output gate $\bm{o}$). Formally, the computational operations to update an LSTM unit at time $t$ are:

[TABLE]

where $[\bm{e}_{t},\bm{c}_{t},\bm{f}^{CharLM}_{t}]$ and $\bm{h}_{t}$ represent the input vector and hidden state, respectively at time $t$. $\sigma$ is the elementwise sigmoid function, and $\odot$ is the elementwise product. $\bm{W}_{i}$, $\bm{W}_{f}$, $\bm{W}_{o}$, $\bm{W}_{m}$ are the weight matrices for the input vector, $\bm{U}_{i}$, $\bm{U}_{f}$, $\bm{U}_{o}$, $\bm{U}_{m}$ are the weight matrices for the hidden state, and $\bm{b}_{i}$, $\bm{b}_{f}$, $\bm{b}_{o}$, $\bm{b}_{m}$ denote the bias vectors.

However, LSTM only considers the information from the past, ignoring future information. To efficiently use contextual information, we can use bidirectional LSTM (BiLSTM). BiLSTM uses a forward LSTM and a backward LSTM for each sequence to obtain two separate hidden states: $\overrightarrow{\bm{h}_{t}}$, $\overleftarrow{\bm{h}_{t}}$, and then the final output at time $t$ is formed by concatenating these two hidden states:

[TABLE]

Therefore, the final output of the BiLSTM layer for the input sentence $S$ can be represented by $\bm{H}=\{\bm{h}_{t}\}^{n}_{t=1}$, where $\bm{H}\in\mathbb{R}^{n\times d}$, and $d$ is the layer size of the BiLSTM layer.

2.3.4 Multihead Self-Attention

Self-attention is a particular case of the attention mechanism, which only requires a single sequence to compute its representation, has been successfully applied to many NLP tasks [43, 44, 32] and shows its superiority in capturing long-range dependency. In SCITE, we adopt the multihead self-attention (MHSA) proposed by Vaswani et al. [32] to learn the dependencies of causalities in the given sentences. Fig. 8 depicts the architecture of the multihead attention mechanism.

Specifically, given $\bm{H}$ as the output of the BiLSTM layer, the multihead attention mechanism first projects the matrix $\bm{H}$ $h$ times with different learned linear projections to matrices: $\bm{H}\bm{W}_{i}^{Q}$, $\bm{H}\bm{W}_{i}^{K}$ and $\bm{H}\bm{W}_{i}^{V}$. where $h$ is the number of heads and parameter matrices $\bm{W}_{i}^{Q}\in\mathbb{R}^{d\times d_{v}}$, $\bm{W}_{i}^{K}\in\mathbb{R}^{d\times d_{v}}$ and $\bm{W}_{i}^{V}\in\mathbb{R}^{d\times d_{v}}$ are projections for the $i$-th head. Then, the attention function is performed in parallel, yielding $n\times d_{v}$-dimensional output values. Finally, all the matrices produced by parallel heads are concatenated, resulting in the final values $\bm{M}$ whose dimension is $n\times(hd_{v})$, where both $h$ and $d_{v}$ are hyperparameters of the self-attention layer. The formulations can be shown as follows:

[TABLE]

Here, the attention function is the “scaled dot-product attention”, which computes the attention scores as follows:

[TABLE]

To fully integrate the information, we concatenate $\bm{H}$ and $\bm{M}$ into matrix $\tilde{\bm{H}}$ and then project $\tilde{\bm{H}}$ with a linear projection to matrix: $\tilde{\bm{H}}\bm{W}$. where weight matrix $\bm{W}\in\mathbb{R}^{(d+hd_{v})k}$ is the parameter of the model to be learned in training and $k$ is the number of distinct tags.

2.3.5 CRF

The conditional random field (CRF) [45] can obtain a globally optimal chain of labels for a given sequence considering the correlations between adjacent tags. In a sequence labeling task, there are usually strong dependencies between the output labels. Therefore, instead of only using RNN to model tagging decisions separately, we adopt BiLSTM-CRF [27] as the backbone of SCIFl to jointly decode labels for the whole sentence.

We use $\bm{P}\in\mathbb{R}^{n\times k}$ as the matrix of scores output by the linear layer, where $\bm{P}_{ij}$ represents the score of the $j^{th}$ label of the $i^{th}$ word within a sentence. For the sentence $S=\{x_{t}\}_{t=1}^{n}$ and a path of tags $y=\{y_{i}\}_{i=1}^{n}$, CRF gives a real-valued score as follows:

[TABLE]

where $\bm{A}$ is the transition matrix, and $\bm{A}_{i,j}$ denotes the score of a transition from tag $i$ to tag $j$. $y_{0}$ and $y_{n}$ are the special tags at the beginning and the end of a sentence, so $\bm{A}$ is a square matrix of size $k+2$. Therefore, the probability for the label sequence $y$ given a sentence $S$ is:

[TABLE]

We now maximize the log-likelihood of the correct tag sequence:

[TABLE]

where $Y_{S}$ represents all possible tag sequences for an input sentence $S$. From the formulation above, we can obtain a valid output sequence. When decoding, the sequence with the maximum score is output by:

[TABLE]

This can be computed using dynamic programming techniques, and we choose the Viterbi algorithm [46] for this decoding.

3 Experiments

3.1 Experimental Settings

3.1.1 Dataset

In the experiment, we evaluate a corpus obtained by extending the annotations of the SemEval 2010 task 8 dataset. [28]. In the original dataset, only one causal triplet in each sentence was annotated. We extend the annotation with the causal triplets not considered by the SemEval annotators; for example, we annotate all of the causal triplets in the sentence in Fig. 2 (more examples are shown in Fig. 3, Fig. 5 and Fig. 6). Specifically, the corpus is composed of 5,236 sentences, of which 1,270 sentences contain at least one causal triplet. The training set consists of 4,450 sentences and contains 1,570 causal triplets. There are 804 sentences in the test set, including 296 causal triplets. Table 2 shows the statistics of six types of causal tags for the dataset.

3.1.2 Evaluation

We use standard precision (P), recall (R) and F1-score (F) as evaluation metrics, which can be calculated by the following formulas:

[TABLE]

where $D$ is the set of all the sentences in the dataset and a predicted causal triplet is regarded as correct if and only if it precisely matches a labeled causal triplet. To obtain comparable and reproducible F1-scores, we follow the advice of Reimers and Gurevych [47] and conduct each experiment 5 times and then report the average results and their standard deviation, as shown in Table 3.

3.1.3 Hyperparameters

The model is implemented by using Keras 333https://github.com/keras-team/keras version 2.2.4. The 300-D word embeddings pretrained by Komninos and Manandhar [40] are employed and kept fixed during the training process. Character embeddings are randomly initialized from a uniform distribution ranging in $[-\sqrt{\frac{3}{dim}},+\sqrt{\frac{3}{dim}}]$, where we set $dim=30$. For the character-level CNN layer, we use a one-layer CNN with 30 filters, and the window size is 3. We use the Flair framework 444https://github.com/zalandoresearch/flair to compute the Flair embeddings. The hidden size of LSTM is set to 256. The parameters $h$ (the number of heads) and $d_{v}$ (the size of each head) of the multihead self-attention mechanism are set to 3 and 8, respectively. We use variational dropout [48] with a dropout rate of 0.5 to regularize our network. To address the exploding gradient problem, we apply gradient normalization [49] with a threshold of 5.0 to the SCITE. The optimization method of the training process is Nadam [50] with a learning rate of 0.001, and we apply a learning rate annealing method such that if the training loss does not fall for more than 10 epochs, this method will halve the learning rate. We let the minibatch size be 16. In the experiments, we perform a grid search and 10-fold cross-validation on the training set to find the optimal hyperparameters. On the test set, we select the optimal model among all 200 epochs with the highest cross-validation F1-score.

3.1.4 Baselines

For a comprehensive comparison, we compare our method against several classic causality extraction methods, which can be divided into two categories: pipeline methods and sequence tagging models based on our causality tagging scheme. The pipeline methods that we use as our baselines are as follows:

Rules+Bayesian: Sorgente et al. [17] performed pattern matching to extract candidate cause-effect pairs based on a set of rules and then used a Bayesian classifier and Laplace smoothing to filter noncausal pairs.

• 2.

CausalNet: Luo et al. [19] proposed causal strength (CS) to measure the causal strength between any two pieces of short texts, integrating necessity causality with sufficiency causality. For comparison, we add the same cause-effect extraction module as Sorgente et al. [17] to their method. We then calculate the CS score of the candidate causal pair and compare it with the threshold $\tau$ ($\tau$ is a tunable hyperparameter). If $CS\left(c,e\right)>\tau$, we conclude that $(c,e)$ is a causal relation; otherwise, $(c,e)$ is an erroneously extracted pair.

The sequence tagging structure used in this paper is divided into CNN-based models and BiLSTM-based models. For the CNN-based models [51], the baselines are as follows:

IDCNN-Softmax: This model uses a deep iterated dilated CNN (IDCNN) architecture to aggregate context from the entire text, which has better capacity than traditional CNN and faster computational speed than LSTM, and then map the output of IDCNN to predict each label independently through a softmax classifier.

• 2.

IDCNN-CRF: This model uses the CRF classifier to maximize the label probability of the complete sentence based on IDCNN. Compared to the softmax classifier, the CRF classifier is more appropriate for tasks with strong output label dependency.

The baselines for the BiLSTM-based models are listed as follows:

BiLSTM-softmax [52]: The model consists of two parts: a BiLSTM encoder and a softmax classifier.

• 2.

BiLSTM-CRF [27]: A classic and popular choice for sequence labeling tasks, which consists of a BiLSTM encoder and a CRF classifier.

• 3.

CLSTM-BiLSTM-CRF [53]: A hierarchical BiLSTM-CRF model that uses character-based representations to implicitly capture morphological features (e.g., prefixes and suffixes) through a character LSTM encoder (CLSTM) and then concatenates the character embeddings and pretrained word embeddings as the input of BiLSTM-CRF.

• 4.

CCNN-BiLSTM-CRF [35]: A similar hierarchical BiLSTM-CRF model uses a character CNN encoder (CCNN) instead of a CLSTM to learn the character-level embeddings.

To further analyze the performance of Flair embeddings transferred into our task, we combine the ELMo [37] and BERT [38], two powerful contextualized word representations, into our task-specific BiLSTM-CRF architecture as the experimental baselines:

ELMo-BiLSTM-CRF: An extension of BiLSTM-CRF in which Peters et al. [37] concatenate pretrained static word embeddings with the ELMo (Embeddings from Language Models) representations and take them as the input of BiLSTM-CRF.

• 2.

BERT-BiLSTM-CRF: A similar extension in which Devlin et al. [38] added pretrained word embeddings and the BERT (bidirectional encoder representations from transformers) representations and used them as the input of BiLSTM-CRF.

• 3.

Flair-BiLSTM-CRF: This model is used as a strong baseline in our work, in which Akbik et al. [31] pretrained word embeddings are concatenated with the Flair embeddings and fed it into the BiLSTM-CRF model. Note that the models using Flair embeddings have achieved the current state-of-the-art results in a range of sequence labeling tasks such as named entity recognition, chunking and part-of-speech tagging [31, 54].

• 4.

Flair+CLSTM-BiLSTM-CRF: A simple extension in which Akbik et al. [31] added task-trained character representations learned from a CLSTM to Flair-BiLSTM-CRF.

3.2 Experimental Results

The performance of different models on the causality extraction is shown in Table 3. The first part is the pipeline methods (from row 2 to row 3). The second part (row 4 to row 5) is the CNN-based sequence tagging method. The third part (row 6 to row 9) is the BiLSTM-based sequence tagging method, and the fourth part (row 10 to row 13) is the sequence tagging method using contextualized word embeddings. Our SCITE model is shown in the final part, where the first row is the result of SCITE based on the proposed tagging scheme and the second row is the result of SCITE based on the general tagging scheme.

Table 3 shows that SCITE outperforms all other models with an F1-score of 0.8455 in the test set. This demonstrates the effectiveness of our proposed method. Furthermore, it also shows that the sequence tagging models are better than pipeline methods.

By comparing the performance of the sequence tagging models on the test set, we can see that the BiLSTM-based models are better than the CNN-based models. The reason for the superior performance of BiLSTM-based models may be that the LSTM layer can more efficiently capture the global word context information and learn semantic representations of causality. In addition, It also shows that the performance of the model drastically improves after feeding contextualized word representations into the BiLSTM-CRF architecture. In particular, the Flair-BiLSTM-CRF achieves the highest improvement of 6.32% over the BiLSTM-CRF compared with the ELMo and BERT (increases of 6.28% and 4.55%, respectively), which indicates that the contextualized character-level word embedding is more suitable for the task of causality extraction.

Moreover, we also find that our proposed causality tagging scheme yields a better result than the general tagging scheme (0.8455 versus 0.7682) under the SCITE architecture, which verifies the effectiveness of our proposed tagging scheme. The general tagging scheme does not contain an “Emb” tag, and thus, the model cannot correctly identify embedded causality. Although the number of embedded causalities is relatively small in the test set, embedded causality plays a crucial role in causality extraction: one error in its identification may affect the correct extraction of multiple triplets, as shown in Fig. 3.

4 Analysis and Discussion

4.1 Error Analysis

In this paper, we focus on extracting all causal triplets from natural language texts, where the accurate identification of tags “C” (cause), “E” (effect) and “Emb” (embedded causality), which represent the semantic roles of causal events, plays a vital role in our task. To perform error analysis, we present a confusion matrix for tags “C” (including “B-C” and “I-C”), “E” (including “B-E” and “I-E”), and “Emb” (including “B-Emb” and “I-Emb”) shown in Fig. 9. We can see that most of the errors are confusion between “C”, “E” and “O”. This confusion may arise due to the problem of insufficient annotated data. Compared with other baselines 555For the convenience of the display, we only show the results of SICFI, Flair-BiLSTM-CRF (the superior of baselines) and BiLSTM-CRF (the classic sequence tagging model). , our model SCITE can better identify “C”, “E”, and “Emb”.

Furthermore, we compare the tagwise performance of our SCITE model with baselines. The comparative results are summarized in Table 4. First, we observe that our model achieves No. 1 in tags “C” (including “B-C” and “I-C”), “E” (including “B-E” and “I-E”), and “Emb” (including “B-Emb” and “I-Emb”) in terms of F1-score. Second, we also notice that the F1-scores are approximately 0.9 except for tag “Emb” because of its low frequency (only 110 instances) in the training set. In particular, it can be seen from the confusion matrix in Fig. 9 that most “Emb” tags in the test set are misidentified as “C” or “E”, which leads to the low recall of “Emb”.

4.2 Ablation Analysis

To investigate the effect of the different components in SCITE (Flair+CCNN-BiLSTM-MHSA-CRF), we also report the results of ablation experiments in Table 5. All parts positively contribute to the performance of the SCITE model.

Specifically, we find that the transferred Flair embeddings provide the most significant improvement. This validates our assumption that the lack of data containing causal triplets in the existing corpus will affect the performance of a neural-based model in causality extraction. Impressively, compared with SCITE without Flair embeddings (SCITE-Flair), the transferred Flair embeddings achieve an improvement of 33.28% in terms of the F1-score in the case of extremely annotated data insufficiency (10% of training data), as shown in Fig. 10. With the help of the transferred contextualized representations, we can not only learn more semantic and syntactic information from the text but also capture word meaning in context to address the polysemous and context-dependent nature of words.

In addition, we also find that the multihead self-attention (MHSA) mechanism can further improve performance, especially when there are no Flair embeddings; the reason is discussed in Section 4.3. Finally, we find that the task-specific character features can also influence the performance of the model by a slight increase when comparing the models with and without the character representations learned from a CCNN.

4.3 Analysis of Multihead Self-Attention

Different from other sequence tagging models, SCITE uses the multihead self-attention mechanism to learn the dependencies between cause and effect. To further analyze the effect of the MHSA, we compute and visualize the F1-score in terms of causality distance (the distance between cause and effect) for the three groups of models:

Group 1: BiLSTM-CRF and BiLSTM-MHSA-CRF;

• 2.

Group 2: Flair-BiLSTM-CRF and Flair-BiLSTM-MHSA-CRF;

• 3.

Group 3: Flair+CCNN-BiLSTM-CRF and SCITE (Flair+CCNN-BiLSTM-MHSA-CRF)

As shown in Fig. 11, we find that the F1-scores decrease with increasing causality distance in all three groups. This validates our assumption that the long-range dependency between cause and effect creates difficulty in causality extraction. In addition, we also see that the performance of models with MHSA is better than that of models without MHSA in arbitrary causality distance, which indicates that the MHSA mechanism plays a crucial role in efficiently enhancing the association between cause and effect. In particular, MHSA significantly improves the performance in terms of the causality distance greater than 10 compared with other cases of shorter causality distance, as shown in Fig. 11a and Fig. 11b.

4.4 Case Study

In Table 6, we list two representative examples to show the advantages and disadvantages of our proposed model. For each case, we show the input sentence and causal triplets contained in the sentence in the first and second row. The remaining rows show the extracted causal triplets of different models 5.

Sentence 1 is the case of simple causality (see Section 2.2.1), in which five causal triplets are waiting for models to extract. We observed that neither the SCITE model nor the other two baseline models could obtain all causal triplets correctly. It seems to be difficult for the models to learn that “superficial or underground water” is a complete semantic unit and the phrase “as well as” may play a key role in connecting two causal components. The reason may be that the sequence tagging models based on our causality tagging scheme require slightly more training data to learn these kinds of causality expression patterns.

Sentence 2 is the case of complex causality (see Section 2.2.2), in which there is an embedded causality and therefore brings difficulty and ambiguity to the causality learning of the model. In this example, only the SCITE can capture all the dependencies between cause and effect and thus precisely extract all three causal triplets when compared with other models.

5 Related Works

In this section, we briefly introduce causality extraction techniques proposed by other researchers, which fall into three categories: 1) approaches that employ pattern matching only, 2) techniques based on the combination of patterns and machine learning, and 3) methods based on deep learning techniques.

5.1 Pattern-Based Methods

Pattern-based methods extract causality through pattern matching using semantic features, lexicon-syntactic features, and self-constructed constraints. For example, Khoo et al. [1] extracted causal knowledge from the Wall Street Journal using linguistic clues and pattern matching. In the domain of the medical abstract, Khoo et al. [11] used graphical patterns to extract causal knowledge from a medical database. Girju and Moldovan [15] extracted causal relations using the syntactic pattern “NP1 causal-verb NP2” with causative verbs and then employed semantic constraints to classify candidates as causal or noncausal. Ittoo and Bouma [16] proposed a causal pair extraction method based on part-of-speech, syntactic analysis, and causality templates. In their work, causality templates were first extracted using causal sentences on Wikipedia, and then they used these templates to extract causal relations in other sentences.

These methods that rely solely on rules for pattern matching often have poor cross-domain applicability and may require extensive domain knowledge in solving problems in a particular area, as well as formulating rules that consume significant amounts of time and effort.

5.2 Methods Based on the Combination of Patterns and Machine Learning

Methods based on the combination of patterns and machine learning techniques mainly treat this task in a pipeline manner. They first extract candidate phrase (or entity, event) pairs that may have causal relations according to templates or some clue words and then classify the candidate causal pairs according to some statistical features or semantic features and grammatical features to filter noncausal pairs. Girju [5] used constraints based on causality trigger words to extract causal relations in English texts and used the C4.5 decision tree to perform classification. Sorgente et al. [17] used predefined templates to extract candidate causal pairs and then used Bayesian classifier and Laplace smoothing to filter noncausal pairs. Zhao et al. [18] proposed a new feature called “causal connectives” by computing the similarity of the syntactic dependency structure of sentences. They run a partial parser to extract candidate noun phrases first and then classified the candidate causal pairs using the restricted hidden naive Bayes learning algorithm in combination with other features, but their method cannot discriminate the causes from the effects. Luo et al. [19] extracted cause-effect terms from large-scale web text corpora using causal cues and then used a new statistical metric-based pointwise mutual information (PMI) to measure causal strength between any two pieces of short texts.

The above methods divide causality extraction into two subtasks: candidate causal pair extraction and relation classification (filtering noncausal pairs). The results of candidate causal pair extraction may affect the performance of relation classification and generate cascading errors. These methods often require considerable human effort and time in feature engineering, relying heavily on the manual selection of textual features, and the hand-selected features are relatively too simple to capture the in-depth semantic information of the context.

5.3 Methods Based on Deep Learning Techniques

Due to the powerful representation learning capabilities of deep neural networks that can effectively capture implicit and ambiguous causal relations, the adoption of deep learning techniques for causality extraction has become a popular choice for researchers in recent years. de Silva et al. [21] used CNN to classify causal relations in the text. Kruengkrai et al. [22] used multicolumn CNN with the background knowledge extracted from noisy texts to classify such commonsense causalities as “smoke cigarettes” $\rightarrow$ “die or lung cancer”. Similarly, Li and Mao [24] proposed a knowledge-oriented CNN that incorporates prior knowledge from lexical knowledge bases for causal relation classification. Martínez-Cámara et al. [23] proposed an LSTM-based model only fed with word embeddings for the task of causality classification. In addition to classifying causality from a common sense reasoning standpoint, Dasgupta et al. [25] and Dunietz et al. [26] also identified the linguistic expressions of causality in the text from a linguistic point of view through deep LSTM-based models.

The main differences between our proposed method and the above methods based on deep learning techniques can be summarized as follows:

Our method aims to automatically extract such common sense causal triplets as c in the text (Fig. 1), not only to classify causal relations or to identify the linguistic expressions of causality.

• 2.

Our method can easily handle multiple causal triplets and embedded causality in the same sentence (Section 2.1 and Section 2.2) without having to divide the sentences into subsentences that contain only one instance of causality and thus generate cascading errors as in Dasgupta et al. [25].

6 Conclusion

In this paper, we formulate causality extraction as a sequence tagging problem and deliver a self-attentive BiLSTM-CRF-based solution for the causality extraction. In particular, we propose SCITE to extract causality in natural language text based on our causality tagging scheme. To alleviate the problem of data insufficiency, we transfer the Flair embeddings trained from a large corpus into our task. In addition, we introduce the multihead self-attention mechanism to learn the dependencies between cause and effect. Experimental results demonstrate the effectiveness of our proposed method. However, the performance of SCITE is still limited to some extent by the insufficiency of high-quality annotated data (Section 4.4).

In future work, we will attempt to solve this problem as follows:

Develop annotated datasets from multiple sources based on existing datasets and our causality tagging scheme. 2. 2.

Combine our method with distant supervision [55] and reinforcement learning [56] to achieve better performance without having to build a high-quality annotated corpus for causality extraction.

7 Acknowledgments

This research is partially supported by the National Natural Science Foundation of China (No. U1711263).