Modeling Contextual Passage Utility for Multihop Question Answering

TL;DR

This paper introduces a lightweight transformer-based method to model the context-dependent utility of passages in multihop QA, improving passage reranking and answer accuracy by considering inter-passage dependencies.

Contribution

It proposes a novel approach to predict passage utility in context, capturing inter-passage relations, which enhances multihop QA performance over existing relevance-based methods.

Findings

Improved reranking of passages in multihop QA tasks.

Enhanced QA accuracy using utility-based passage scoring.

Effective modeling of inter-passage dependencies.

Abstract

Multihop Question Answering (QA) requires systems to identify and synthesize information from multiple text passages. While most prior retrieval methods assist in identifying relevant passages for QA, further assessing the utility of the passages can help in removing redundant ones, which may otherwise add to noise and inaccuracies in the generated answers. Existing utility prediction approaches model passage utility independently, overlooking a critical aspect of multihop reasoning: the utility of a passage can be context-dependent, influenced by its relation to other passages - whether it provides complementary information or forms a crucial link in conjunction with others. In this paper, we propose a lightweight approach to model contextual passage utility, accounting for inter-passage dependencies. We fine-tune a small transformer-based model to predict passage utility scores for multihop QA. We leverage the reasoning traces from an advanced reasoning model to capture the order in which passages are used to answer a question and obtain synthetic training data. Through comprehensive experiments, we demonstrate that our utility-based scoring of retrieved passages leads to improved reranking and downstream QA performance compared to relevance-based reranking methods.