TL;DR
This study investigates what makes reading comprehension questions difficult, finding that passage source and readability do not significantly impact difficulty, but diverse passage selection influences question types and reasoning requirements.
Contribution
It provides empirical insights into factors affecting question difficulty and reasoning types, guiding better selection of passages for benchmark datasets.
Findings
Passage source and readability do not significantly affect difficulty
Diverse passages lead to a variety of question types and reasoning
Logical reasoning is more common in technical passages
Abstract
For a natural language understanding benchmark to be useful in research, it has to consist of examples that are diverse and difficult enough to discriminate among current and near-future state-of-the-art systems. However, we do not yet know how best to select text sources to collect a variety of challenging examples. In this study, we crowdsource multiple-choice reading comprehension questions for passages taken from seven qualitatively distinct sources, analyzing what attributes of passages contribute to the difficulty and question types of the collected examples. To our surprise, we find that passage source, length, and readability measures do not significantly affect question difficulty. Through our manual annotation of seven reasoning types, we observe several trends between passage sources and reasoning types, e.g., logical reasoning is more often required in questions written for…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
