Dynamic Transfer Learning for Named Entity Recognition

TL;DR

This paper introduces Dynamic Transfer Networks (DTN), a novel gated architecture that adaptively learns optimal parameter sharing for low-resource named entity recognition, especially in clinical texts, eliminating the need for extensive hyperparameter search.

Contribution

The paper proposes DTN, a new model that automatically learns parameter sharing schemes for transfer learning in NER, improving performance without exhaustive search.

Findings

DTN outperforms baseline models in clinical NER tasks.

DTN achieves comparable results to optimized transfer schemes with a single training.

The approach reduces the need for exponential hyperparameter search.

Abstract

State-of-the-art named entity recognition (NER) systems have been improving continuously using neural architectures over the past several years. However, many tasks including NER require large sets of annotated data to achieve such performance. In particular, we focus on NER from clinical notes, which is one of the most fundamental and critical problems for medical text analysis. Our work centers on effectively adapting these neural architectures towards low-resource settings using parameter transfer methods. We complement a standard hierarchical NER model with a general transfer learning framework consisting of parameter sharing between the source and target tasks, and showcase scores significantly above the baseline architecture. These sharing schemes require an exponential search over tied parameter sets to generate an optimal configuration. To mitigate the problem of exhaustively searching for model optimization, we propose the Dynamic Transfer Networks (DTN), a gated architecture which learns the appropriate parameter sharing scheme between source and target datasets. DTN achieves the improvements of the optimized transfer learning framework with just a single training setting, effectively removing the need for exponential search.

Tables

Table 1. (A) Medication (i2b2) to TTP (i2b2) (10%)

Model	Precision	Recall	${\text{F}}_1$
Baseline	55.20	48.25	51.47
Highest Performance TTN
IIS	75.79	74.43	75.10
HIH	75.65	74.29	74.96
III	75.42	74.34	74.87
Lowest Performance TTN
HSS	74.92	73.71	74.31
SSI	75.65	72.83	74.21
SSH	74.65	73.29	73.96
Avg.	75.47	73.69	74.57 $\pm$ 0.24
DTN	75.65	73.61	74.46
DTN (HS)	75.83	74.09	74.95

Table 2. Table 2: Overview of i2b2, affiliate, and newswire datasets.

	Med	TTP	Affiliate	CoNLL	Onto
Tags	25	13	37	17	73
Notes	252	426	1000	1393	3,637
Tokens	336K	416K	1.5M	301K	2M

Table 3. OntoNotes to CoNLL (10%)

Highest Performance TTN
Model	Precision	Recall	${\text{F}}_1$
HIH	82.74	81.23	81.98
SIS	81.98	80.86	81.42
Avg.	80.66	79.47	80.06 $\pm$ 1.04
DTN (HS)	82.45	81.78	82.12

Table 4. Table 4: Gate activations are averaged across all tokens from input, for experiment two. These results look at a gate choosing between hard and soft sharing (Eq. 1). A low value indicates the gate favored hard sharing, whereas a value closer to 1.0 1.0 1.0 favors soft sharing.

Component	Char Enc	Word Enc	Decoder
Medication Name	0.64	0.91	0.77
Form	0.88	0.99	0.18
Dosage	0.69	0.99	0.26
Frequency	0.81	0.98	0.22
Overall	0.65	0.32	0.82

Equations

$$P(\mathbf{y}|\mathbf{x},\theta )=\prod _{t=1}^{T}P(y_t|x_t,y_{1:t-1},\theta )$$

$$o_t=\text{LSTM}(o_{t-1},[h_t||{\hat{y}}_{t-1}])$$

$${\hat{y}}_t=\text{softmax}(\mathbf{W}{o}_t+b^s),$$

$$\theta ={\theta }_{source}\cup {\theta }_{target}\cup {\theta }_{shared}$$

$${\mathcal{L}}_{share}=||{\theta }_{source}-{\theta }_{target}|{|}_2^2$$

$$g_1=\sigma ({\mathbf{Q}}^{⊺}{h}_{\text{t-soft}}+{\mathbf{R}}^{⊺}{h}_{\text{t-hard}}+{\mathbf{S}}^{⊺}{a}_{\text{target}}+b_{g_1})$$

$$o_{\text{shared}}=(1-g_1)h_{\text{t-hard}}+g_1\cdot h_{\text{t-soft}}$$

$$g_2=\sigma ({\mathbf{T}}^{⊺}{h}_{\text{ind}}+{\mathbf{U}}^{⊺}{o}_{\text{shared}}+{\mathbf{V}}^{⊺}{a}_{\text{target}}+b_{g_2})$$

$$o_{\text{target}}=(1-g_2)h_{\text{ind}}+g_2\cdot o_{\text{shared}}$$

$$o_{\text{source}}=h_{\text{s-hard}}+h_{\text{s-soft}}$$

$${\mathcal{L}}_{CE}={\mathcal{L}}_{\text{target}}+{\mathcal{L}}_{\text{source}}$$

$$\mathcal{L}={\mathcal{L}}_{CE}+\lambda {\mathcal{L}}_{share}$$

Full text

Introduction

Natural Language Processing (NLP) applications have been significantly enhanced through advances in neural architecture design. Tasks such as machine translation, summarization (?), language modeling (?), and information extraction have all achieved state of the art systems using deep neural networks, however with a caveat. These applications require large datasets to generalize well, and naturally sparse domains benefit less from such robust systems. One such domain is medical data. Specifically, clinical notes, the free text contents of electronic health records (EHR), have limited availability due to the delicate nature of their content. Privacy concerns prevent the public release of clinical notes, and furthermore de-identification, and annotation is a lengthy and costly process.

We are interested in Named Entity Recognition (NER)(?) within low-resource areas such as medical domains. NER is a sequence labeling task similar to part of speech (POS) tagging, and text chunking. For medical data, NER is an important application as an information extraction tool for downstream tasks such as entity linking (?) and relation extraction (?). Medical text has challenges that are unique to its domain as well. Clinicians will often use shorthand or abbreviations to produce patient release notes, with irregular grammar. This gives the text a significantly less formal grammatical structure than standard NER datasets which often focus on newswire data (?). There is also a high degree of variance across sub-domains, which can be attributed to the degree of specialty hospital departments have (e.g. cardiology vs radiology). While certain medical jargon, and hospital procedure may be invariant of specialty; diseases, treatments, and medications will likely be correlated under these specific sub-domains. Building an NER system that can learn to generalize well across these is therefore quite difficult, and building individual systems for sub-domains is equally arduous due to the lack of data. Therefore, we turn towards transfer learning to diminish the effects of data accessibility, and to leverage overlapping representation across sub-domains.

Transfer learning (?) is a learning paradigm that seeks to enhance performance of a target task with knowledge from a source task. This can take several forms: as pretraining, where a model is first trained for a source task and then some or all weights are used for initialization of the target task; or in place of feature engineering using word embeddings (?; ?), a popular approach for most NLP tasks. We look towards parameter sharing methods (?) to transfer overlapped representation from source to target task, when both are NER.

Parameter sharing schemes feature tied weights between layers of a neural network across several tasks. Finding useful configurations of parameter sharing has been the focus of several recent papers (?; ?; ?; ?; ?). As model depth increases the number of possible architectures grows exponentially, and it becomes difficult to exhaustively search through all configurations to choose the best model. We show that these design choices are a learnable component of the model, and propose a new transfer learning architecture; a generalized neural model which dynamically updates independent and shared components achieving similar scores of models which have been fully tuned.

Our contributions are as follows:

• •

We propose the Tunable Transfer Network (TTN). A framework which unifies existing parameter sharing techniques into a single model. This network compartmentalizes all components of our baseline architecture. Furthermore, we fully explore three degrees of parameter sharing with this system: hard, soft, and independent. This architecture allows searching for the parameter sharing scheme that best suits the transfer learning setting.

• •

Addressing the large search space problem in TTN, we propose Dynamic Transfer Networks (DTN), a dynamic gated architecture that learns the appropriate parameter sharing between source and target tasks across multiple sharing schemes. DTN mitigate the issue of exhaustive architecture exploration, while achieving similar performance of the optimized tunable network.

• •

We present a thorough empirical analysis of parameter sharing for low resource named entity recognition on medical data. We also demonstrate DTN’s effectiveness on a non-medical dataset achieving best results in such settings.

We will first introduce related work as background for NER as well as transfer learning, followed by our proposed architecture, system setup, and dataset information. We conclude with our findings on low resource settings in both medical and non-medical domains.

Related Work

NER models achieved their recent success with neural architectures. In 2016 several works (?; ?; ?) proposed hierarchical sequence to sequence deep learning frameworks. The models enjoyed RNN, or CNN encoders, but generally utilized conditional random fields (CRF) as decoders. Many subsequent works have focused on fine-tuning for speed or parameter size, while keeping this model design at a high level.

Transfer learning for both NER, and other NLP tasks has also been extensively studied. Here, we will look towards generic models, with more of a focus on those which targeted the medical domain. (?) leverage unsupervised pretraining in the form of forward and backward language modeling to initialize most of the parameters of an NER architecture. Their model was also evaluated on medical data and although the performance increased with pre-training, the evaluation showed low recall from unseen entities. (?) were among the first to explore parameter sharing with the general neural NER architecture. The authors explored training for NER with other sequence tagging tasks, across multiple languages. Continuing their work they also correlated task similarity with the number of shared layers in a model (?). These architectures generally employed a forked design where feature representations were shared at the lower layers (closer towards the input) and split towards multiple outputs. For example, tasks in the same language, and with similar labels would share a larger number of layers, whereas sequencing in English and Spanish, regardless of the output space may share only the input embeddings. The approach of sharing lower level layers was also used for semantic parsing (?), and co training language models (?). In the latter only a character level encoder was shared between tasks, and highway units control feature transfer to downstream components. We employ a similar technique by gating features from multiple inputs at the same layer. Shared label embedding layers have also shown favorable results (?; ?). For multiple tasks a single softmax is used with masking for non-task labels. The shared embeddings better promote label synergy.

Directly sharing parameters has been widely used, however transfer learning schemes have utilized a soft sharing paradigm as well, where model parameters or outputs are constrained to a similar space. Most similar to our work, (?) use two types of constraints to promote shared representations of overlapping output distributions, as well as latent representations. This work minimizes parameter difference of the CRFs which is derived as the Kullback Leibler divergence upper bound minimization of the target task against the source across overlapping labels from both tasks. Additionally they constrain the model to produce similar latent representations for tokens with the same tag. This work is also applied towards NER across several medical sub-domains. Using soft sharing transfer learning for summarization, (?) jointly train three generative models. Their work was also novel to not have the forked design, in that both the input and output layers were independent. The same authors used a similar architecture with more ablation on sharing for sentence simplification (?).

The parameter sharing architectures discussed here all suffer from the need to exhaustively search for the best architecture, if it is even performed. Our approach mitigates this procedure by allowing the model to decide what form of parameter sharing it should employ at various layers, and is able to do this during a single training session.

Our model also draws inspiration from pointer networks (?; ?). Pointer networks has shown great performance in assisting generative models augment their output vocabulary distribution with knowledge of the input sequence. Our work, however, uses this technique to transfer the signal across several parameter sharing components.

Models

We first present a standard neural framework for named entity recognition. We expand on that architecture by building the Tunable Transfer Network (TTN), to incorporate transfer learning options to each layer. Finally, we introduce the Dynamic Transfer Network (DTN), as a trainable transfer learning framework extending the TTN.

Named Entity Recognition Architecture

A sequence tagging problem such as NER can be formulated as maximizing the conditional probability distribution over tags $\mathbf{y}$ given an input sequence $\mathbf{x}$, and model parameters $\theta$.

[TABLE]

$T$ is the length of the sequence, and $y_{1:t-1}$ are tags for the previous words. The architecture we use as a foundation is that of (?; ?; ?), and while we provide a brief overview of this model we refer the reader to any of these works for architectural insights. The model consists of three main components: the (i) character and (ii) word encoders, and the (iii) decoder/tagger.

Encoders

Given an input sequence $\mathbf{x}\in\mathbb{N}^{T}$ whose coordinates indicate the words in the input vocabulary, we first encode the character level representation for each word. For each $x_{t}$ the corresponding sequence $\mathbf{c}^{(t)}\in\mathbb{R}^{L\times e_{c}}$ of character embeddings is fed into an encoder. Here $L$ is the length of a given word and $e_{c}$ is the size of the character embedding. The character encoder employs two Long Short Term Memory (LSTM) (?) units which produce $\overrightarrow{h^{(t)}_{1:l}}$, and $\overleftarrow{h^{(t)}_{1:l}}$, the forward and backward hidden representations respectively, where $l$ is the last timestep in both sequences. We concatenate the last timestep of each of these as the final encoded representation, $h_{c}^{(t)}=[\overrightarrow{h^{(t)}_{l}}||\overleftarrow{h^{(t)}_{l}}]$, of $x_{t}$ at the character level.

The output of the character encoder is concatenated with a pre-trained word embedding (?), $m_{t}=[h_{c}^{(t)}||\text{emb}_{word}(x_{t})]$, which is used as the input to the word level encoder. Using learned character embeddings alongside word embeddings has shown to be useful for learning word level morphology, as well as mitigating loss of representation for out-of-vocabulary words. Similar to the character encoder we use a bidirectional LSTM (BiLSTM) (?) to encode the sequence at the word level. The word encoder does not lose resolution, meaning the output at each timestep is the concatenated output of both word LSTMs, $h_{t}=[\overrightarrow{h_{t}}||\overleftarrow{h_{t}}]$.

Decoder and Tagger

Finally the concatenated output of the word encoder is used as input to the decoder, along with the label embedding of the previous timestep. During training we use teacher forcing (?) to provide the gold standard label as part of the input.

[TABLE]

[TABLE]

where $\mathbf{W}\in\mathbb{R}^{d\times n}$, $d$ is the number of hidden units in the decoder LSTM, and $n$ is the number of tags. The model is trained in an end to end fashion using a standard cross-entropy objective.

In most of the recent NER literature the focus has been on optimizing accuracy and speed by investigating different neural mechanisms for the three components (?; ?). Both convolutional and recurrent networks have been explored for the encoders, with either conditional random fields (CRF), or single directional RNNs employed as the decoder/tagger. Since extensive work has been done on this front we fix the design settings and focus only transfer learning while using this common NER architecture. As described, we use BiLSTM units for both encoders, and a uni-directional LSTM as the decoder.

Tunable Transfer Network

The tunable transfer network extends to the components from the previous sections. Here we focus on how best to benefit from transfer learning with respect to each layer. To reformulate the architecture from this perspective the model will always train on two tasks, henceforth labeled as source and target. Model parameters will be decomposed as:

[TABLE]

Source and target parameters are updated by training examples from their respective datasets, while shared parameters receive updates from both tasks. Updates for parameters will depend on the batch focus, meaning for a given forward pass of the model a batch will contain data from either the source or target task. During training we shuffle the batches among tasks to allow the model to alternate randomly between them.

We now describe the parameter sharing architectures:

• •

Independent parameters, Figure 1 (left). Relative to the component, the network performs no transfer learning across the two parameters. For some layers the model performs best when no shared knowledge exists. This is generally applied to input or output layers.

• •

Hard parameter sharing, Figure 1 (right). The parameters of both components reference the same set of weights, and each task in turn updates the same weights.

• •

Soft parameter sharing, Figure 1 (center). Individual weights are given to both source and target components, however if this sharing paradigm is present in the model we add an additional segment to the objective:

[TABLE]

Here, we minimize the $l_{2}$ distance between parameters as a form of regularization. Soft sharing loosely couples corresponding parameters to one another while allowing for more freedom than hard sharing, hence allowing different tasks to choose what sections of their parameters space to share.

The sharing paradigms from TTN intuitively represent the relatedness of the latent representation of the two tasks for a given component. Since these are tunable hyperparameters of the architecture, we optimize the model by finding the best configuration of sharing. Optimizing this involves training $O(M^{N})$ unique models, where $M$ is the number of sharing schemes, and $N$ the number of tunable layers. Another problem with the current setup is that for some output distributions the target task may already exhibit high confidence in labels, and introducing a sharing scheme may in fact induce a bias towards the source task.

Dynamic Transfer Network

Searching across different model architectures motivates us to build a model similar to Figure 2 which is robust enough to overcome an exponential search of model architecture and achieve similar results compared to the tuned TTN model. As mentioned above, being able to tune model architecture is costly, and it is preferable to allow the system to learn how much of a representation to exploit from the source task vs. feedback from its own labels.

Therefore we propose to use the Dynamic Transfer Network (DTN), where gating mechanisms similar to highway units(?), or pointer generators (?), control the signal strength from a shared and non-shared component of the network. We use these gates to choose the best representation between hard and soft sharing, and then between sharing and independent parameters. This multi-staged gating is similar to the layered pointers used by (?).

The architecture of DTN is illustrated in Figure 3. To begin, our source and target inputs both pass through their respective RNNs which employ soft (center), and hard (right) sharing, in parallel. The target and source RNNs take as input $a_{\text{target}}$, and $a_{\text{source}}$ respectively. This produces two latent representations for both: $h_{\text{t-soft}}$, $h_{\text{s-soft}}$, $h_{\text{t-hard}}$, and $h_{\text{s-hard}}$, where t, and s denote target and source. We then determine which sharing mechanism was more useful for the target task using a gating function:

[TABLE]

[TABLE]

We also used an independent (left) RNN, to produce a third latent representation for the target, $h_{\text{ind}}$. Our second gating function takes this, as well as the output of the first gated function as input.

[TABLE]

[TABLE]

The final result is a combined representation of the target task as input to subsequent layers. For both gates, $\sigma$ is the sigmoid function, and $\mathbf{Q}$, $\mathbf{R}$, $\mathbf{S}$, $\mathbf{T}$, $\mathbf{U}$, $\mathbf{V}$, $b_{g_{1}}$, and $b_{g_{2}}$ are trainable parameters. Since our task focuses on how best to adapt the layer towards the target task, the source hidden representations are simply added element-wise to produce:

[TABLE]

The final loss for a network using DTN (Figure 2) has the weighted soft sharing regularization objective, along with the cross entropy loss of both tasks.

[TABLE]

[TABLE]

TTN has a similar objective, however not all configurations will contain $\mathcal{L}_{share}$.

Inference

Both the TTN, and DTN use only parameters for the target task during evaluation and inference. For example the soft sharing component does not make use of the source parameters at this time.

Experimental Setup

Datasets

Our work utilizes four main corpora where we employ a tagging scheme that follows an inside, outside, begin, end and singleton (IOBES) format. We use the public datasets from the 2009 and 2010 i2b2 challenges for medication (Med) (?), and “test, treatment, problem” (TTP) entity extraction.

The second dataset is obtained through an affiliate, and it is annotated similar to the i2b2 medication challenge. Both of the above datasets contain free-text release notes, which have been de-identified.

Additionally, we explore non-medical, newswire data: CoNLL 2003 English (?) and OntoNotes 5.0 English (?).

Model Settings

Word, character and tag embeddings are 100, 25, and 50 dimensions respectively. Word embeddings are initialized using GloVe (?), while character and tag embeddings are learned from scratch. Character, and word encoders have 50, and 100 hidden units respectively. Decoder LSTM has a hidden size of 50. Dropout is used after every LSTM, as well as for word embedding input. We use Adam (?) as an optimizer. Our model is built using MXNet (?).

DTN Hard-Soft

We also evaluate a simplified version of the DTN presented in the previous section. This model, denoted as DTN (HS) under results, learns the best transfer learning setting between soft coupling and hard sharing. This model retains the first gate (Eq. 1 and 2) from the architecture and uses $o_{\text{shared}}$ as the final target signal for each component.

Experiments

Our models are trained until convergence, and we use the development set of the target task to evaluate performance for early stopping. We focus on transfer learning in three settings. The first setting uses only the i2b2 dataset, where the target task is TTP, and the source task is medication. The second set of experiments uses our affiliate medication data as a target, with i2b2 medication data as the source. The third task is non-medical, and uses CoNLL 2003 as the target, with OntoNotes 5.0 as the source. The first and third setting also allows for reproducible performance since the data is publicly available. We evaluate the performance of our models on $10\%$ of the total target dataset for the first TL setting, and $5\%$ for the second setting. For non-medical setting, we used $10\%$ of the total target dataset. The source dataset is not reduced in any of the experiments. Development and test set are also kept the original size. The baseline follows the construction of the architecture described in the first section of modeling.

Results

We analyze our results from multiple perspectives. We first demonstrate the effectiveness of parameter sharing for low resource settings for both experiments in the medical domain. We also examine model performance across various data percentages to showcase the uniform performance of DTN models. Furthermore, we explore the gating values across layers to investigate model behavior for the dynamic architecture which suggests why gating can imbibe the characteristics of the best model which varies depending upon the relatedness of the source and target tasks. We report precision, recall, and macro $\text{F}_{1}$ on the target data test set.

Transfer Learning Performance

The test set results on all medical data are reported in Table 1. For the tunable network, we show results for six models (three best, and three worst), as well as the average result across all 27 configurations (three components, and three sharing schemes).

For the first setting (Table 1A), there is on average a 36.66$\%$ $\text{F}_{1}$ gain over the baseline model which indicates that the system greatly benefited from transfer learning. Similarly there was a 11.21$\%$ increase for TTN across the medication only tasks (Table 1B). Notably all settings of the tunable model yielded a large margin in performance over both baselines. More consequential, however, is the range of performance among the tunable models. We observed high variance in the first task with the lowest $\text{F}_{1}$ score (soft-soft-hard) of $73.96$ versus the highest $75.10$ (indep-indep-soft). The second task had a similar gap of 1 $\text{F}_{1}$ point between high and low performers. These results validate the need to search for the best architecture for parameter sharing.

DTN

In general DTN performed well, and more intriguing was the capability of DTN (HS), as it always triumphed over its more complex counterpart. For the first task, the dynamic model achieved a score of $74.46$, and DTN (HS) outperformed all but the best two. Although slightly less robust in the second TL task, the DTN placed within the $63$’rd percentile of TTN. DTN (HS) similarly bested all but the top two models. Model performance is even more commendable for the newswire data. In Table 3, we see a boost over the best performing TTN, with DTN (HS) placing two standard deviations above the TTN average. This validates our hypothesis that the given TTN architectures are not exhaustive to produce the best results and dynamic nature of our model helps in encapsulating the best model setting for a given problem.

Our model provides competitive performance with significantly reduced training time. Moreover, we show that the DTN is consistent in performance while TTN configurations exhibit variability across training conditions. Figure 5 illustrates this phenomenon. We chose the best and worst TTN settings for a particular low resource (10%) setting (from i2b2 medication to TTP) and we see that the rankings are not as tightly coupled when we re-execute the experiments with more (20%) or fewer (5%) training samples. Another example can be seen from Table 1, where the best performing model, SSI, for medication data sets was one of the worst performing models in case of i2b2 dataset.

Gating

We take a closer look at output across a sequence in Figure 4. We compare the output of the DTN against better performing TTN models to show how the model adapts when others fail. The illustration is indicative that the model does not rely on a particular gating scheme consistently. Instead we observe the changes in gating across a sequence, where the model relies on multiple learning schemes for a given token.

We further analyzed the contributions of DTN between the different sharing schemes. Upon a closer inspection of the output layer gates as shown in Table 4, we observe significant variance among parameter sharing across different tag types. The parameter sharing for tags depends on the relatedness of the target and source tags. For example, Form is not present in the i2b2 (source) dataset. We discern that the decoder sharing scheme for the Form tag prefers hard sharing thus smaller value, as it can not leverage much information from the soft sharing scheme. Overall we observe interesting insights, where a parameter sharing scheme depends on the tag type as well as temporality thereby making RNN more robust to the sensitivity of the data.

Conclusion

In this paper we have shown that tuning a transfer learning architecture in low resource settings will allow for a more efficient architecture. We further mitigated this exponential search process by introducing the dynamic transfer network to learn the best transfer learning settings for a given hierarchical architecture. We showed the generalization of this model across different named entity recognition datasets. For future work, we plan to explore our model on other sequential problems such as translation, summarization, chat bots as well as explore more advanced gating schemes.