Towards Inference-Oriented Reading Comprehension: ParallelQA

TL;DR

This paper introduces ParallelQA, a new method for testing neural reading comprehension models on inference-based questions using parallel passages, revealing their limited generalization ability.

Contribution

The paper proposes ParallelQA, a novel question formulation strategy to evaluate inference-oriented reasoning in neural MRC models, highlighting their shortcomings.

Findings

Neural models struggle with inference-based questions in ParallelQA.

Existing models do not generalize well to inference-oriented tasks.

ParallelQA exposes limitations of current MRC systems.

Abstract

In this paper, we investigate the tendency of end-to-end neural Machine Reading Comprehension (MRC) models to match shallow patterns rather than perform inference-oriented reasoning on RC benchmarks. We aim to test the ability of these systems to answer questions which focus on referential inference. We propose ParallelQA, a strategy to formulate such questions using parallel passages. We also demonstrate that existing neural models fail to generalize well to this setting.