On the Effectiveness of Compact Biomedical Transformers

TL;DR

This paper introduces six lightweight biomedical transformer models created through knowledge distillation and continual learning, achieving comparable performance to larger models while significantly reducing resource requirements.

Contribution

The paper presents six novel compact biomedical transformers that match larger models' performance, developed via knowledge distillation and continual learning techniques.

Findings

Models perform on par with BioBERT-v1.1 on biomedical tasks.

All models are publicly available for research and practical use.

Significant reduction in computational resources needed for biomedical NLP.

Abstract

Language models pre-trained on biomedical corpora, such as BioBERT, have recently shown promising results on downstream biomedical tasks. Many existing pre-trained models, on the other hand, are resource-intensive and computationally heavy owing to factors such as embedding size, hidden dimension, and number of layers. The natural language processing (NLP) community has developed numerous strategies to compress these models utilising techniques such as pruning, quantisation, and knowledge distillation, resulting in models that are considerably faster, smaller, and subsequently easier to use in practice. By the same token, in this paper we introduce six lightweight models, namely, BioDistilBERT, BioTinyBERT, BioMobileBERT, DistilBioBERT, TinyBioBERT, and CompactBioBERT which are obtained either by knowledge distillation from a biomedical teacher or continual learning on the Pubmed dataset via the Masked Language Modelling (MLM) objective. We evaluate all of our models on three biomedical tasks and compare them with BioBERT-v1.1 to create efficient lightweight models that perform on par with their larger counterparts. All the models will be publicly available on our Huggingface profile at https://huggingface.co/nlpie and the codes used to run the experiments will be available at https://github.com/nlpie-research/Compact-Biomedical-Transformers.