Momentum Contrastive Pre-training for Question Answering

TL;DR

This paper introduces MCROSS, a momentum contrastive pre-training method that aligns cloze-like and natural questions to improve extractive question answering models, showing significant gains on benchmark datasets.

Contribution

The paper proposes a novel contrastive pre-training framework that enhances transferability from cloze-like to natural questions in extractive QA.

Findings

Improved performance on three QA benchmarks.

Effective in both supervised and zero-shot settings.

Outperforms baseline pre-training methods.

Abstract

Existing pre-training methods for extractive Question Answering (QA) generate cloze-like queries different from natural questions in syntax structure, which could overfit pre-trained models to simple keyword matching. In order to address this problem, we propose a novel Momentum Contrastive pRe-training fOr queStion anSwering (MCROSS) method for extractive QA. Specifically, MCROSS introduces a momentum contrastive learning framework to align the answer probability between cloze-like and natural query-passage sample pairs. Hence, the pre-trained models can better transfer the knowledge learned in cloze-like samples to answering natural questions. Experimental results on three benchmarking QA datasets show that our method achieves noticeable improvement compared with all baselines in both supervised and zero-shot scenarios.