Enhancing Biomedical Text Summarization and Question-Answering: On the Utility of Domain-Specific Pre-Training

TL;DR

This paper demonstrates that in biomedical text summarization, a three-step fine-tuning approach with general-domain pre-training can outperform domain-specific pre-training, especially with limited in-domain data.

Contribution

It introduces a novel three-step fine-tuning method that leverages general-domain pre-training for biomedical summarization, challenging the assumption that domain-specific pre-training is always superior.

Findings

General-domain pre-training followed by task-specific fine-tuning improves performance.

In-domain pre-training does not always provide advantages in biomedical summarization.

A large language model without domain-specific pre-training can excel in biomedical text generation.

Abstract

Biomedical summarization requires large datasets to train for text generation. We show that while transfer learning offers a viable option for addressing this challenge, an in-domain pre-training does not always offer advantages in a BioASQ summarization task. We identify a suitable model architecture and use it to show a benefit of a general-domain pre-training followed by a task-specific fine-tuning in the context of a BioASQ summarization task, leading to a novel three-step fine-tuning approach that works with only a thousand in-domain examples. Our results indicate that a Large Language Model without domain-specific pre-training can have a significant edge in some domain-specific biomedical text generation tasks.