Robustifying Multi-hop QA through Pseudo-Evidentiality Training

TL;DR

This paper introduces a novel pseudo-evidentiality training method for multi-hop question answering that enhances model robustness by learning to identify supporting evidence without requiring costly annotations.

Contribution

It proposes a new approach to learn evidentiality through counterfactual analysis, reducing annotation costs while improving reasoning robustness in multi-hop QA models.

Findings

The method improves accuracy on HotpotQA.

It enhances robustness against reasoning biases.

It achieves comparable or better results than evidence-annotated training.

Abstract

This paper studies the bias problem of multi-hop question answering models, of answering correctly without correct reasoning. One way to robustify these models is by supervising to not only answer right, but also with right reasoning chains. An existing direction is to annotate reasoning chains to train models, requiring expensive additional annotations. In contrast, we propose a new approach to learn evidentiality, deciding whether the answer prediction is supported by correct evidences, without such annotations. Instead, we compare counterfactual changes in answer confidence with and without evidence sentences, to generate "pseudo-evidentiality" annotations. We validate our proposed model on an original set and challenge set in HotpotQA, showing that our method is accurate and robust in multi-hop reasoning.