Enhancing Pre-Trained Generative Language Models with Question Attended Span Extraction on Machine Reading Comprehension

TL;DR

This paper introduces the QASE module, which enhances pre-trained generative language models for machine reading comprehension by improving their span extraction ability, outperforming some large models without extra computational costs.

Contribution

The paper presents a novel QASE module integrated during fine-tuning that significantly boosts generative models' extractive performance in MRC tasks, surpassing some large language models.

Findings

QASE improves generative models' accuracy on MRC datasets.

Enhanced models outperform GPT-4 in few-shot settings.

Performance gains are achieved without increased computational costs.

Abstract

Machine Reading Comprehension (MRC) poses a significant challenge in the field of Natural Language Processing (NLP). While mainstream MRC methods predominantly leverage extractive strategies using encoder-only models such as BERT, generative approaches face the issue of out-of-control generation -- a critical problem where answers generated are often incorrect, irrelevant, or unfaithful to the source text. To address these limitations in generative models for MRC, we introduce the Question-Attended Span Extraction (QASE) module. Integrated during the fine-tuning phase of pre-trained generative language models (PLMs), QASE significantly enhances their performance, allowing them to surpass the extractive capabilities of advanced Large Language Models (LLMs) such as GPT-4 in few-shot settings. Notably, these gains in performance do not come with an increase in computational demands. The efficacy of the QASE module has been rigorously tested across various datasets, consistently achieving or even surpassing state-of-the-art (SOTA) results, thereby bridging the gap between generative and extractive models in extractive MRC tasks.