Clozer: Adaptable Data Augmentation for Cloze-style Reading Comprehension

TL;DR

Clozer introduces a flexible sequence-tagging method for data augmentation in cloze-style reading comprehension, significantly improving task performance by better extracting answer spans without relying on heuristics.

Contribution

The paper presents Clozer, a novel extendable sequence-tagging approach for cloze answer extraction that enhances task-adaptive pre-training for reading comprehension.

Findings

Clozer outperforms existing methods and the oracle in MRC tasks.

Clozer effectively recognizes gold answers independently of heuristics.

Significant performance gains in multiple-choice cloze-style MRC tasks.

Abstract

Task-adaptive pre-training (TAPT) alleviates the lack of labelled data and provides performance lift by adapting unlabelled data to downstream task. Unfortunately, existing adaptations mainly involve deterministic rules that cannot generalize well. Here, we propose Clozer, a sequence-tagging based cloze answer extraction method used in TAPT that is extendable for adaptation on any cloze-style machine reading comprehension (MRC) downstream tasks. We experiment on multiple-choice cloze-style MRC tasks, and show that Clozer performs significantly better compared to the oracle and state-of-the-art in escalating TAPT effectiveness in lifting model performance, and prove that Clozer is able to recognize the gold answers independently of any heuristics.