Transfer Learning for Causal Sentence Detection

TL;DR

This paper investigates the effectiveness of transfer learning methods like ELMO and BERT for detecting causal sentences, finding benefits mainly in small datasets and limited gains with larger datasets.

Contribution

It introduces a transfer learning approach for causal sentence detection and evaluates its performance across different dataset sizes, including a new biomedical dataset.

Findings

Transfer learning improves performance on small datasets.

Larger datasets lead to performance plateaus, reducing transfer learning benefits.

Transfer learning has limited impact on large datasets.

Abstract

We consider the task of detecting sentences that express causality, as a step towards mining causal relations from texts. To bypass the scarcity of causal instances in relation extraction datasets, we exploit transfer learning, namely ELMO and BERT, using a bidirectional GRU with self-attention (BIGRUATT) as a baseline. We experiment with both generic public relation extraction datasets and a new biomedical causal sentence detection dataset, a subset of which we make publicly available. We find that transfer learning helps only in very small datasets. With larger datasets, BIGRUATT reaches a performance plateau, then larger datasets and transfer learning do not help.

Tables

Table 1. Table 1: Precision, recall, f1 , auc on the four publicly available datasets, averaged over 10 repetitions, with standard deviations ( ± plus-or-minus \pm ). Next to each dataset name, we show in brackets the total causal and non-causal sentences that we used. The best results are shown in bold. The best auc results are also shown in gray background. In the auc columns, stars indicate statistically significant ( p ≤ 0.05 𝑝 0.05 p\leq 0.05 ) differences compared to bigruatt .

Dataset (causal:non-causal)

SemEval (1,325 : 2,500)

CausalTB (244 : 500)

EventSL (77 : 200)

BioCausal-Small (1,113 : 887)

Model

F1

P

R

AUC

F1

P

R

AUC

F1

P

R

AUC

F1

P

R

AUC

LR ($n$-grams)

76.22

88.67

66.83

87.50

36.36

​100.00

22.22

65.02

42.86

​100.00

27.27

73.55

77.49

73.91

81.44

87.65

BIGRUATT

90.64 $\pm$0.70

93.96 $\pm$1.71

87.59 $\pm$1.52

96.57 $\pm$0.32

69.98 $\pm$3.58

67.04 $\pm$5.16

73.89 $\pm$6.60

74.38 $\pm$4.16

63.65 $\pm$10.12

70.09 $\pm$7.47

60.91 $\pm$17.28

70.36 $\pm$9.84

85.97 $\pm$0.90

83.57 $\pm$2.03

88.62 $\pm$2.69

93.91 $\pm$0.88

BIGRUATT+ELMO

92.81 $\pm$0.78

94.45 $\pm$0.94

91.26 $\pm$1.77

97.03 $\pm$1.44

75.08 $\pm$4.20

81.29 $\pm$5.43

70.28 $\pm$6.81

82.06* $\pm$3.59

66.55 $\pm$7.82

77.47 $\pm$5.05

59.09 $\pm$10.17

77.31 $\pm$4.84

87.32 $\pm$0.78

89.46 $\pm$2.34

85.39 $\pm$2.50

94.95 $\pm$0.33

BERT+LR

91.55 $\pm$0.53

86.62 $\pm$1.16

97.09 $\pm$0.67

96.94 $\pm$2.25

80.55 $\pm$3.62

71.17 $\pm$6.02

93.33 $\pm$3.33

82.26* $\pm$3.41

72.35 $\pm$5.36

62.44 $\pm$8.21

87.17 $\pm$4.58

78.15* $\pm$9.48

85.64 $\pm$0.61

78.87 $\pm$1.16

93.71 $\pm$1.54

90.75 $\pm$3.69

BERT+BIGRUATT

91.45 $\pm$0.59

86.80 $\pm$1.28

96.63 $\pm$0.60

97.61 $\pm$0.29

80.06 $\pm$2.94

74.52 $\pm$4.46

86.94 $\pm$5.56

84.27* $\pm$1.71

73.09 $\pm$5.27

66.15 $\pm$7.86

83.64 $\pm$10.60

84.17* $\pm$4.04

85.87 $\pm$0.88

79.43 $\pm$1.53

93.47 $\pm$1.08

93.75 $\pm$0.45

Table 2. Table 2: Results on BioCausal-Large (7,562 : 5,780).

Model	F1	P	R	AUC
LR ($n$-grams)	79.21	76.75	81.82	86.54
BIGRUATT	85.84 $\pm$0.36	84.28 $\pm$0.66	87.47 $\pm$0.75	93.71 $\pm$0.15
BIGRUATT+ELMO	86.77 $\pm$0.52	87.46 $\pm$1.29	86.12 $\pm$1.49	94.64* $\pm$0.26
BERT+LR	87.33 $\pm$0.47	82.11 $\pm$0.89	93.27 $\pm$0.67	92.77 $\pm$1.57
BERT+BIGRUATT	87.09 $\pm$0.34	81.70 $\pm$0.58	93.25 $\pm$0.40	94.70 $\pm$0.24

Equations

$$H^f$$

$$H^b$$

$$H$$

$${\tilde{a}}_i=u_{att}\cdot h_i,a_i=\text{softmax}({\tilde{a}}_i;{\tilde{a}}_1,\dots ,{\tilde{a}}_n)$$

$$s=\sum _{i=1}^n{a}_i{h}_i,p=\sigma (u_p\cdot s+b_p)$$

Full text

1 Introduction

A wide range of biomedical questions, from what causes a disease to what drug dosages should be recommended and which side effects might be triggered, center around detecting particular causal relationships between biomedical entities. Causality, therefore, has long been a focus of biomedical research, e.g., in medical diagnostics Rizzi (1994), pharmacovigilance Agbabiaka et al. (2008), and epidemiology Karhausen (2000). The most common way to detect causal relationships is by carrying highly controlled randomized controlled trials, but it is also possible to mine evidence from observational studies and meta-analyses Ward and Johnson (2008), where information is often expressed in natural language (e.g., journal articles or clinical study reports).

In natural language processing (nlp), causality detection is often viewed as a type of relation extraction, where the goal is to determine which relations (e.g., part-whole, content-container, cause-effect), if any, hold between two entities in a text Hendrickx et al. (2009), using deep learning in most recent works Bekoulis et al. (2018); Zhang et al. (2018). The same view of causality detection is typically adopted in biomedical nlp Cohen and Demner-Fushman (2014); Li and Mao (2019).

Existing relation extraction datasets, however, contain few causal instances, which may not allow relation extraction methods to learn to infer causality reliably. Note that causality can be expressed in many ways, from using explicit lexical markers (e.g., “smoking causes cancer”) to markers that do not always express causality (e.g., “heavy smoking led to cancer” vs. “the nurse led the patient to her room”) to no explicit markers (“she was infected by a virus and admitted to a hospital”). Also, existing relation extraction datasets contain sentences from generic, not biomedical documents.

In this paper, we focus on detecting causal sentences, i.e., sentences conveying at least one causal relation. This is a first step towards mining causal relations from texts. Once causal sentences have been detected, computationally more intensive relation extraction methods can be used to identify the exact entities that participate in the causal relations and their roles (cause, effect). To bypass the scarcity of causal instances in relation extraction datasets, we exploit transfer learning, namely elmo Peters et al. (2018) and bert Devlin et al. (2018), comparing against a bidirectional gru with self-attention Cho et al. (2014); Bahdanau et al. (2015). We experiment with generic public relation extraction datasets and a new larger biomedical causal sentence detection dataset, a subset of which we make publicly available.111We cannot provide the entire biomedical dataset, because it is used to develop commercial products. We report, however, results for both the entire biomedical dataset and the publicly available subset. Unlike recently reported results in other nlp tasks Peters et al. (2018); Devlin et al. (2018); Peters et al. (2019), we find that transfer learning helps only in datasets with hundreds of training instances. When a few thousands of training instances are available, bigruatt reaches a performance plateau (both in generic and biomedical texts), then increasing the size of the dataset or employing transfer learning does not improve performance. We believe this is the first work to (a) focus on causal sentence detection as a binary classification task, (b) consider causal sentence detection in both generic and biomedical texts, and (c) explore the effect of transfer learning in this task.

2 Methods

bigruatt: Our baseline model is a bidirectional gru (bigru) with self-attention (bigruatt) Cho et al. (2014); Bahdanau et al. (2015), a classifier that has been reported to perform well in short text classification Pavlopoulos et al. (2017); Chalkidis et al. (2019). The model views each sentence as the sequence $\left<e_{1},\dots,e_{n}\right>$ of its word embeddings (Fig. 1). We use word2vec embeddings Mikolov et al. (2013) pre-trained on approx. (a) 3.5 billion tokens from pubmed texts McDonald et al. (2018)222http://nlp.cs.aueb.gr/software.html or (b) 100 billion tokens from Google News.333https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM The bigru computes two lists $H_{f},H_{b}$ of hidden states, reading the word embeddings left to right and right to left, respectively. The corresponding elements of $H_{f},H_{b}$ are then concatenated to form the output $H$ of the bigru:

[TABLE]

where f, b indicate the forward and backward directions, $e_{i}\in\mathbb{R}^{d_{e}}$, $h^{f}_{i},h^{b}_{i}\in\mathbb{R}^{d_{h}}$, and ‘;’ denotes concatenation.444In our experiments, $d_{h}=128$; $d_{e}$ is 300 for Google News and 200 for biomedical embeddings; $d_{e}$ increases by 1,024 when elmo is added. We also tried lstms Hochreiter and Schmidhuber (1997), but performance was similar. A linear attention computes an attention score $a_{i}\in\mathbb{R}$ for each element $h_{i}$ of $H$:

[TABLE]

where $u_{att}\in\mathbb{R}^{2\times d_{h}}$ and $\cdot$ is the dot product. A sentence embedding $s$, representing the entire sentence, is then formed as the weighted (by the attention scores) sum of the elements of $H$ and is passed to a logistic regression (lr) layer to estimate the probability $p$ that a sentence is causal:

[TABLE]

where $u_{p}\in\mathbb{R}^{2\times d_{h}}$, $b_{p}\in\mathbb{R}$, and $\sigma$ is the sigmoid function. We use cross-entropy loss, the Adam optimizer Kingma and Ba (2015), and dropout layers Srivastava et al. (2014) before and after the bigru (Fig. 1). Word embeddings are not updated.

bigruatt+elmo: elmo Peters et al. (2018) produces word embeddings by passing the input text (in our case, a sentence) to a pre-trained stacked bidirectional lstm language model (lm). It then uses a linear combination of the states of the lm (from the different layers of the stacked lstm) at each word position to produce the corresponding word embedding. Like word2vec, elmo (its lm) is pre-trained on large corpora. However, elmo maps occurrences of the same word to possibly different embeddings, depending on context. Furthermore, it uses cnns LeCun et al. (1989) to produce the initial word embeddings (that are fed to the lm) from word characters, alleviating the problem of out-of-vocabulary words. bigruatt+elmo is the same as bigruatt, except that the embedding of each word is now the concatenation of its word2vec and elmo embeddings. We do not update the parameters of elmo and the word embeddings when training bigruatt+elmo. We used the original pre-trained elmo of Peters et al. (2018).555https://allennlp.org/elmo For biomedical sentences, we also experimented with an elmo model pre-trained on pubmed texts, but performance was very similar as with the original elmo.

bert+lr: bert Devlin et al. (2018) is a model based on Transformers Vaswani et al. (2017), pre-trained on large corpora to predict (a) masked words from their left and right contexts, and (b) the next sentence. For a new nlp task, a task-specific layer is added on top of a pre-trained bert model. The extra layer is trained jointly with bert on task-specific data (in our case, a causal sentence detection dataset), a process that also fine-tunes the parameters of the pre-trained bert for the new task. In bert+lr, we add a logistic regression (lr) layer on top of bert, which estimates the probability that the input sentence is causal. The lr layer is fed with the embedding of the ‘classification’ token, which bert also produces for each sentence. We used the pre-trained ‘base’ bert model of Devlin et al. (2018), which we fine-tuned jointly with the lr layer. For biomedical sentences, we also experimented with biobert Lee et al. (2019), a bert model pre-trained on biomedical texts, but performance was very similar.

bert+bigruatt: Common practice Devlin et al. (2018); Peters et al. (2019) is to combine bert with very shallow task-specific models, usually only an lr layer. To explore if deeper task-specific models can yield improved performance, we replaced the lr layer of bert+lr with bigruatt, leading to bert+bigruatt. This is the same as bigruatt, but uses the context-aware word embeddings that bert produces at its top layer as the input to bigruatt, instead of word2vec embeddings. Again, we use the ‘base’ pre-trained bert model of Devlin et al. (2018), and we fine-tune the entire bert+bigruatt network on causal sentence detection datasets.

lr (n-grams): A plain lr classifier with tf-idf $n$-gram features (word $n$-grams, $n=1,2,3$).666We used the lr code of scikit-learn (https://scikit-learn.org/). For all other methods, we used our own PyTorch implementations (https://pytorch.org/), with the bert api of https://github.com/huggingface/pytorch-pretrained-BERT .

3 Datasets

SemEval-2010 (Task 8): This dataset contains 10,674 samples, of which 1,325 causal Hendrickx et al. (2009). Each sample is a sentence annotated with a pair of entities and the type of their relationship. Since we are only interested in causality, we treat sentences with a Cause-Effect relationship as causal, and all the others as non-causal. Causal-TimeBank (CausalTB): In this dataset Mirza et al. (2014), we identified causal sentences using c-signal (causal signal) and clink (causal link) tags, discarding causal relationships between entities from different sentences, following Li and Mao (2019). The resulting dataset contains 2,470 sentences, of which 244 are causal.

Event StoryLine (EventSL): In this dataset Caselli and Vossen (2017), we detected causal sentences by examining the causes and caused_by attributes in the plot_link tags, again following Li and Mao (2019). Again, we discarded causal relationships between entities from different sentences. The resulting dataset contains 4,107 sentences, of which 77 are causal.

BioCausal: The full biomedical causal detection dataset we developed (BioCausal-Large) contains 13,342 sentences from pubmed, of which 7,562 causal. Each sentence was annotated by a single annotator familiar with biomedical texts.777The average inter-annotator agreement on a sample of 300 sentences was 79.36%. Cohen’s Kappa was 0.56. The publicly available subset (BioCausal-Small) contains 2,000 sentences, of which 1,113 causal.888BioCausal-Small is available at https://archive.org/details/CausalySmall.

Li and Mao (2019) report that SemEval-2010 contains a large number of causal samples with explicit causal markers; by contrast, CausalTB and EventSL contain more complex causal relations with no explicit clues. BioCausal includes causal sentences both with and without explicit clues.

SemEval-2010, CausalTB, and EventSL are highly imbalanced, with the vast majority of sentences being non-causal. To prevent a high bias towards the non-causal class, in our experiments we randomly selected 2500, 500, 200 non-causal sentences respectively, discarding the rest. The resulting causal to non-causal ratios (Table 1) are, thus, roughly 1:2 (SemEval, CausalTB) or 1:3 (EventSL). By contrast, the BioCausal (Large and Small) datasets are roughly balanced. All five datasets were then split into train (70%), validation (15%) and test (15%) subsets, maintaining the same ratio between the two classes in the three subsets.

4 Experimental results

Tables 1–2 report our experimental results. For each neural model, we performed 10 repetitions (with different random seeds) and report averages and standard deviations. For completeness, we show precision, recall, f1, and area under the precision-recall curve (auc), though auc scores are the main ones to consider, since they examine performance at multiple classification thresholds; the other measures are computed only for a particular threshold, which was 0.5 in our experiments.

Focusing on auc scores, bigruatt outperforms the simpler lr with $n$-grams by a wide margin, with the exception of EventSL, which is probably too small for the capacity of bigruatt.999Indeed bigruatt overfits the training set of EventSL. The precision of lr is perfect on CausalTB and EventSL, at the expense of very low recall, suggesting that lr learned perfectly few high-precision $n$-grams in those datasets. Transfer learning (elmo, bert) improves the auc of bigruatt by a wide margin in the two smallest datasets (CausalTB, EventSL), which contain only hundreds of instances, and the auc differences from bigruatt are statistically significant (stars in Table 1), except for bigruatt+elmo in EventSL.101010We performed two-tailed Approximate Randomization tests Dror et al. (2018), $p\leq 0.05$, with 10k iterations, randomly swapping in each iteration 50% of the decisions (over all tested sentences) across the two compared methods. When testing statistical significance, for each method we use the repetition (among the 10) with the best validation f1 score. However, in the other three datasets which contain thousands of instances, the auc differences between transfer learning and plain bigruatt are small, with no statistical significance in most cases. Also, the auc scores of all methods on BioCausal-Large are close to those on BioCausal-Small, despite the fact that BioCausal-Large is approx. seven times larger. Similar observations can be made by looking at the f1 scores.

It seems that causal sentence detection, at least with the neural methods we considered, reaches a plateau with few thousands of training sentences both in generic and biomedical texts; then increasing the dataset size or employing transfer learning does not help. The latter finding is not in line with previously reported results Peters et al. (2018); Devlin et al. (2018); Peters et al. (2019), where elmo and bert were found to improve performance in most nlp tasks without studying, however, the effect of dataset size. Furthermore, bert+bigruatt consistently performed better than bert+lr in auc (but not in f1), which casts doubts on the practice of adding only shallow task-specific models to bert.111111We note, however, that the auc difference between bert+lr and bert+bigruatt is statistically significant ($p\leq 0.05$) only in BioCausal-Large. bigruatt+elmo is competitive in auc to bert+bigruatt (with the exception of EventSL). Mainly comprised of sentences with simple explicit causal statements Li and Mao (2019), SemEval expectedly demonstrated the best classification performance across datasets.

5 Related and Future Work

Recent work on (causal) relation extraction uses lstms Zhang et al. (2017) or cnns Li and Mao (2019), assuming however that the spans of the two entities (cause, effect) are known. A notable exception is the model of Bekoulis et al. (2018), which jointly infers the spans of the entities and their relationships. Such finer relation extraction methods, however, are computationally more expensive than our causal sentence detection methods, especially when they involve parsing Zhang et al. (2018). We plan to consider pipelines where computationally cheaper causal sentence detection components will first detect sentences likely to express causality, and then finer relation extraction components will pinpoint the entities, the type of causality (e.g., up-regulate), and entity roles.

Appendix

Appendix A Hyper-parameters

Batch sizes of 128, 32, 16, 32 and 256 were used for Semeval, CausalTB, EventSL, BioCausal-Small and BioCausal-Large, respectively, for all neural models. Trainable parameters were initialized using the default PyTorch initialization methods except from self-attention weights where the method of Glorot and Bengio (2010) was used.

bigru and bigruatt+elmo were trained for 100 epochs using Adam Kingma and Ba (2015) with initial learning rate 2e-3 , $\beta_{1}/\beta_{2}=0.9/0.999$ and $eps=1e^{-8}$. The learning rate was decayed linearly every 20 epochs as $lr_{new}\leftarrow lr_{prev}\cdot 0.75$. Gradients were clipped using a clip norm threshold of 0.25. The gru’s hidden size was set to 128 and its depth to 1. A dropout of 0.5 was applied to the input and output connections of the bigru encoder. Validation f1 was checked periodically in order to keep the model’s checkpoint with the best validation performance.

bert and bert+bigruatt used the bert-base uncased pre-trained model, which has 12 layers, 768 hidden size, 12 attention heads, and 110M parameters. For both models the entire network was fine-tuned for 10 epochs using Adam with a very small learning rate of 2e-5, $\beta_{1}/\beta_{2}=0.9/0.999$, $eps=1e^{-6}$, L2 weight decay of 0.01 and linear warmup of 0.1. A dropout of 0.1 was applied to all bert-specific layers. For bert+bigruatt, an additional dropout of 0.5 was applied to the input and output connections of its bigru encoder. Similarly to bigru and bigruatt+elmo, the hidden size of the gru was set to 128 and its depth to 1.