MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension

TL;DR

The MRQA 2019 shared task evaluated the ability of reading comprehension systems to generalize across diverse datasets, using unified data formats and multiple training strategies, with the best system significantly outperforming the baseline.

Contribution

This paper introduces a unified framework for evaluating generalization in reading comprehension across multiple datasets and reports on the performance of various innovative system approaches.

Findings

Best system achieved 72.5 F1 score, 10.7 points above baseline.

Multiple strategies like data sampling and adversarial training improved results.

Unified dataset format facilitated cross-dataset evaluation.

Abstract

We present the results of the Machine Reading for Question Answering (MRQA) 2019 shared task on evaluating the generalization capabilities of reading comprehension systems. In this task, we adapted and unified 18 distinct question answering datasets into the same format. Among them, six datasets were made available for training, six datasets were made available for development, and the final six were hidden for final evaluation. Ten teams submitted systems, which explored various ideas including data sampling, multi-task learning, adversarial training and ensembling. The best system achieved an average F1 score of 72.5 on the 12 held-out datasets, 10.7 absolute points higher than our initial baseline based on BERT.