Pre-training data selection for biomedical domain adaptation using journal impact metrics

TL;DR

This study investigates whether selecting biomedical pre-training data based on journal impact metrics improves language model performance, finding that impact-based pruning is ineffective but reducing data size does not harm performance.

Contribution

The paper introduces a simple impact metric-based data selection approach for biomedical domain adaptation and evaluates its effectiveness compared to using full datasets.

Findings

Impact-based pruning is not effective for data selection.

Using fewer abstracts with the same training steps maintains performance.

Pre-training data size reduction does not necessarily decrease model quality.

Abstract

Domain adaptation is a widely used method in natural language processing (NLP) to improve the performance of a language model within a specific domain. This method is particularly common in the biomedical domain, which sees regular publication of numerous scientific articles. PubMed, a significant corpus of text, is frequently used in the biomedical domain. The primary objective of this study is to explore whether refining a pre-training dataset using specific quality metrics for scientific papers can enhance the performance of the resulting model. To accomplish this, we employ two straightforward journal impact metrics and conduct experiments by continually pre-training BERT on various subsets of the complete PubMed training set, we then evaluate the resulting models on biomedical language understanding tasks from the BLURB benchmark. Our results show that pruning using journal impact metrics is not efficient. But we also show that pre-training using fewer abstracts (but with the same number of training steps) does not necessarily decrease the resulting model's performance.