World Knowledge in Multiple Choice Reading Comprehension

TL;DR

This paper investigates how multiple choice reading comprehension systems use world knowledge to answer questions without passage context, and proposes metrics to assess and control this reliance for better test design.

Contribution

It introduces information-theory based metrics to evaluate the extent of world knowledge used by systems, aiding test designers in creating fair questions.

Findings

Questions answerable by world knowledge are often answerable by humans without context.

Proposed metrics can identify questions that rely heavily on world knowledge.

Metrics help ensure questions require passage understanding rather than shortcuts.

Abstract

Recently it has been shown that without any access to the contextual passage, multiple choice reading comprehension (MCRC) systems are able to answer questions significantly better than random on average. These systems use their accumulated "world knowledge" to directly answer questions, rather than using information from the passage. This paper examines the possibility of exploiting this observation as a tool for test designers to ensure that the use of "world knowledge" is acceptable for a particular set of questions. We propose information-theory based metrics that enable the level of "world knowledge" exploited by systems to be assessed. Two metrics are described: the expected number of options, which measures whether a passage-free system can identify the answer a question using world knowledge; and the contextual mutual information, which measures the importance of context for a given question. We demonstrate that questions with low expected number of options, and hence answerable by the shortcut system, are often similarly answerable by humans without context. This highlights that the general knowledge 'shortcuts' could be equally used by exam candidates, and that our proposed metrics may be helpful for future test designers to monitor the quality of questions.