Fine-Tuning Pretrained Language Models With Label Attention for Biomedical Text Classification

TL;DR

This paper introduces a transformer-based biomedical text classifier that incorporates label descriptions through a label attention module during fine-tuning, improving classification performance on medical datasets.

Contribution

It presents a novel label attention mechanism integrated into pretrained language models for biomedical text classification, leveraging label descriptions for better accuracy.

Findings

Outperforms vanilla PTMs on two public datasets

Achieves state-of-the-art results in biomedical text classification

Demonstrates the effectiveness of label-aware fine-tuning

Abstract

The massive scale and growth of textual biomedical data have made its indexing and classification increasingly important. However, existing research on this topic mainly utilized convolutional and recurrent neural networks, which generally achieve inferior performance than the novel transformers. On the other hand, systems that apply transformers only focus on the target documents, overlooking the rich semantic information that label descriptions contain. To address this gap, we develop a transformer-based biomedical text classifier that considers label information. The system achieves this with a label attention module incorporated into the fine-tuning process of pretrained language models (PTMs). Our results on two public medical datasets show that the proposed fine-tuning scheme outperforms the vanilla PTMs and state-of-the-art models.