Counterfactual Multihop QA: A Cause-Effect Approach for Reducing Disconnected Reasoning

TL;DR

This paper introduces a causal-effect based counterfactual approach to improve multi-hop question answering by reducing disconnected reasoning, leading to more accurate multi-hop reasoning and better performance on HotpotQA.

Contribution

It proposes a novel causal-effect framework with counterfactual inference to explicitly model and reduce disconnected reasoning in multi-hop QA models.

Findings

Achieves 5.8% higher Supp_s score on HotpotQA

Effectively disentangles disconnected reasoning from true multi-hop reasoning

Demonstrates significant improvement over baseline models

Abstract

Multi-hop QA requires reasoning over multiple supporting facts to answer the question. However, the existing QA models always rely on shortcuts, e.g., providing the true answer by only one fact, rather than multi-hop reasoning, which is referred as $\textit{disconnected reasoning}$ problem. To alleviate this issue, we propose a novel counterfactual multihop QA, a causal-effect approach that enables to reduce the disconnected reasoning. It builds upon explicitly modeling of causality: 1) the direct causal effects of disconnected reasoning and 2) the causal effect of true multi-hop reasoning from the total causal effect. With the causal graph, a counterfactual inference is proposed to disentangle the disconnected reasoning from the total causal effect, which provides us a new perspective and technology to learn a QA model that exploits the true multi-hop reasoning instead of shortcuts. Extensive experiments have conducted on the benchmark HotpotQA dataset, which demonstrate that the proposed method can achieve notable improvement on reducing disconnected reasoning. For example, our method achieves 5.8% higher points of its Supp$_s$ score on HotpotQA through true multihop reasoning. The code is available at supplementary material.