Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets

TL;DR

This paper introduces the BLUE benchmark for biomedical NLP, evaluates BERT and ELMo models on it, and finds that domain-specific BERT models outperform others across various biomedical and clinical tasks.

Contribution

The paper creates a new benchmark for biomedical NLP and systematically evaluates popular language models, highlighting the effectiveness of domain-specific pre-training.

Findings

BERT models trained on biomedical data outperform general models.

The BLUE benchmark covers diverse biomedical and clinical NLP tasks.

Pre-trained models and datasets are publicly available for research.

Abstract

Inspired by the success of the General Language Understanding Evaluation benchmark, we introduce the Biomedical Language Understanding Evaluation (BLUE) benchmark to facilitate research in the development of pre-training language representations in the biomedicine domain. The benchmark consists of five tasks with ten datasets that cover both biomedical and clinical texts with different dataset sizes and difficulties. We also evaluate several baselines based on BERT and ELMo and find that the BERT model pre-trained on PubMed abstracts and MIMIC-III clinical notes achieves the best results. We make the datasets, pre-trained models, and codes publicly available at https://github.com/ncbi-nlp/BLUE_Benchmark.