Enhancing Clinical Information Extraction with Transferred Contextual Embeddings

TL;DR

This paper demonstrates that BERT-based models, when properly fine-tuned, significantly improve clinical information extraction tasks, achieving state-of-the-art results on nursing handover data.

Contribution

The study shows effective transfer of BERT models to biomedical NLP tasks, achieving new state-of-the-art performance with minimal domain-specific adaptation.

Findings

BERT-based models outperform previous IE models on clinical data.

The macro-average F1 score improved from 0.416 to 0.438.

Transfer learning with BERT is effective in health-related NLP applications.

Abstract

The Bidirectional Encoder Representations from Transformers (BERT) model has achieved the state-of-the-art performance for many natural language processing (NLP) tasks. Yet, limited research has been contributed to studying its effectiveness when the target domain is shifted from the pre-training corpora, for example, for biomedical or clinical NLP applications. In this paper, we applied it to a widely studied a hospital information extraction (IE) task and analyzed its performance under the transfer learning setting. Our application became the new state-of-the-art result by a clear margin, compared with a range of existing IE models. Specifically, on this nursing handover data set, the macro-average F1 score from our model was 0.438, whilst the previous best deep learning models had 0.416. In conclusion, we showed that BERT based pre-training models can be transferred to health-related documents under mild conditions and with a proper fine-tuning process.