REPT: Bridging Language Models and Machine Reading Comprehension via Retrieval-Based Pre-training

TL;DR

REPT introduces a retrieval-based pre-training method that enhances evidence extraction capabilities in language models, significantly improving performance on multi-sentence reasoning MRC tasks through self-supervised tasks and consistent retrieval strategies.

Contribution

The paper proposes a novel retrieval-based pre-training approach with self-supervised tasks to improve evidence extraction in language models for MRC.

Findings

Significant performance gains on five MRC datasets.

Enhanced evidence extraction without explicit supervision.

Improved reasoning across multiple sentences.

Abstract

Pre-trained Language Models (PLMs) have achieved great success on Machine Reading Comprehension (MRC) over the past few years. Although the general language representation learned from large-scale corpora does benefit MRC, the poor support in evidence extraction which requires reasoning across multiple sentences hinders PLMs from further advancing MRC. To bridge the gap between general PLMs and MRC, we present REPT, a REtrieval-based Pre-Training approach. In particular, we introduce two self-supervised tasks to strengthen evidence extraction during pre-training, which is further inherited by downstream MRC tasks through the consistent retrieval operation and model architecture. To evaluate our proposed method, we conduct extensive experiments on five MRC datasets that require collecting evidence from and reasoning across multiple sentences. Experimental results demonstrate the effectiveness of our pre-training approach. Moreover, further analysis shows that our approach is able to enhance the capacity of evidence extraction without explicit supervision.