MuSiQue: Multihop Questions via Single-hop Question Composition

TL;DR

MuSiQue introduces a systematic bottom-up approach to create challenging multihop QA datasets that require genuine reasoning, addressing shortcuts and disconnected reasoning in existing benchmarks.

Contribution

We propose a novel bottom-up methodology for constructing multihop questions with controlled properties, resulting in the MuSiQue dataset that emphasizes true multihop reasoning.

Findings

MuSiQue-Ans contains 25K questions with 2-4 hops, increasing difficulty.

Models show a 3x larger gap in performance compared to existing datasets.

Disconnection-based shortcuts are significantly reduced, making the dataset more robust.

Abstract

Multihop reasoning remains an elusive goal as existing multihop benchmarks are known to be largely solvable via shortcuts. Can we create a question answering (QA) dataset that, by construction, \emph{requires} proper multihop reasoning? To this end, we introduce a bottom-up approach that systematically selects composable pairs of single-hop questions that are connected, i.e., where one reasoning step critically relies on information from another. This bottom-up methodology lets us explore a vast space of questions and add stringent filters as well as other mechanisms targeting connected reasoning. It provides fine-grained control over the construction process and the properties of the resulting $k$-hop questions. We use this methodology to create MuSiQue-Ans, a new multihop QA dataset with 25K 2-4 hop questions. Relative to existing datasets, MuSiQue-Ans is more difficult overall (3x increase in human-machine gap), and harder to cheat via disconnected reasoning (e.g., a single-hop model has a 30 point drop in F1). We further add unanswerable contrast questions to produce a more stringent dataset, MuSiQue-Full. We hope our datasets will help the NLP community develop models that perform genuine multihop reasoning.