How to Pre-Train Your Model? Comparison of Different Pre-Training Models for Biomedical Question Answering

TL;DR

This paper compares different pre-training models for biomedical question answering, showing that open domain question answering models outperform reading comprehension models when fine-tuned on small datasets.

Contribution

It provides an empirical comparison of pre-training approaches, highlighting the effectiveness of open domain question answering models for biomedical QA tasks.

Findings

Open domain QA pre-training outperforms reading comprehension pre-training.

Pre-training improves model performance on small biomedical datasets.

Open domain models are more suitable for biomedical question answering.

Abstract

Using deep learning models on small scale datasets would result in overfitting. To overcome this problem, the process of pre-training a model and fine-tuning it to the small scale dataset has been used extensively in domains such as image processing. Similarly for question answering, pre-training and fine-tuning can be done in several ways. Commonly reading comprehension models are used for pre-training, but we show that other types of pre-training can work better. We compare two pre-training models based on reading comprehension and open domain question answering models and determine the performance when fine-tuned and tested over BIOASQ question answering dataset. We find open domain question answering model to be a better fit for this task rather than reading comprehension model.