AMMU : A Survey of Transformer-based Biomedical Pretrained Language Models

TL;DR

This survey comprehensively reviews transformer-based biomedical pretrained language models, covering their foundational concepts, taxonomy, challenges, and future research directions in the biomedical NLP domain.

Contribution

It provides the first detailed taxonomy and analysis of biomedical transformer-based PLMs, summarizing core concepts, models, challenges, and open issues.

Findings

Various transformer-based BPLMs have been developed for biomedical NLP.

Pretraining methods and tasks vary across models, impacting performance.

Open issues include data scarcity and model interpretability.

Abstract

Transformer-based pretrained language models (PLMs) have started a new era in modern natural language processing (NLP). These models combine the power of transformers, transfer learning, and self-supervised learning (SSL). Following the success of these models in the general domain, the biomedical research community has developed various in-domain PLMs starting from BioBERT to the latest BioELECTRA and BioALBERT models. We strongly believe there is a need for a survey paper that can provide a comprehensive survey of various transformer-based biomedical pretrained language models (BPLMs). In this survey, we start with a brief overview of foundational concepts like self-supervised learning, embedding layer and transformer encoder layers. We discuss core concepts of transformer-based PLMs like pretraining methods, pretraining tasks, fine-tuning methods, and various embedding types specific to biomedical domain. We introduce a taxonomy for transformer-based BPLMs and then discuss all the models. We discuss various challenges and present possible solutions. We conclude by highlighting some of the open issues which will drive the research community to further improve transformer-based BPLMs.