Span Selection Pre-training for Question Answering

TL;DR

This paper introduces Span Selection Pre-Training (SSPT), a novel pre-training task for Transformers that improves question answering performance by aligning pre-training with reading comprehension, achieving state-of-the-art results on multiple datasets.

Contribution

The paper proposes SSPT, a new pre-training task that enhances language understanding for question answering by selecting answer spans from relevant passages during training.

Findings

Achieves SOTA on Natural Questions with 3 F1 points improvement.

Improves HotpotQA answer and supporting fact prediction.

More effective with limited training data, enhancing learning efficiency.

Abstract

BERT (Bidirectional Encoder Representations from Transformers) and related pre-trained Transformers have provided large gains across many language understanding tasks, achieving a new state-of-the-art (SOTA). BERT is pre-trained on two auxiliary tasks: Masked Language Model and Next Sentence Prediction. In this paper we introduce a new pre-training task inspired by reading comprehension to better align the pre-training from memorization to understanding. Span Selection Pre-Training (SSPT) poses cloze-like training instances, but rather than draw the answer from the model's parameters, it is selected from a relevant passage. We find significant and consistent improvements over both BERT-BASE and BERT-LARGE on multiple reading comprehension (MRC) datasets. Specifically, our proposed model has strong empirical evidence as it obtains SOTA results on Natural Questions, a new benchmark MRC dataset, outperforming BERT-LARGE by 3 F1 points on short answer prediction. We also show significant impact in HotpotQA, improving answer prediction F1 by 4 points and supporting fact prediction F1 by 1 point and outperforming the previous best system. Moreover, we show that our pre-training approach is particularly effective when training data is limited, improving the learning curve by a large amount.