GenSco: Can Question Decomposition based Passage Alignment improve Question Answering?

TL;DR

GenSco leverages question decomposition and passage alignment with dual LLMs to improve multi-hop question answering accuracy efficiently, reducing hallucinations and enhancing answer relevance.

Contribution

The paper introduces GenSco, a novel passage selection method based on question decomposition guided by an auxiliary LLM, improving multi-hop QA performance.

Findings

Achieved 15.1 and 5.9 points improvements in Exact Match scores on MuSiQue and 2WikiMultiHop datasets.

Demonstrated cost-effective and efficient passage selection with a single invocation of the generator LLM.

Outperformed existing baselines in multi-hop question answering accuracy.

Abstract

Retrieval augmented generation (RAG) with large language models (LLMs) for Question Answering (QA) entails furnishing relevant context within the prompt to facilitate the LLM in answer generation. During the generation, inaccuracies or hallucinations frequently occur due to two primary factors: inadequate or distracting context in the prompts, and the inability of LLMs to effectively reason through the facts. In this paper, we investigate whether providing aligned context via a carefully selected passage sequence leads to better answer generation by the LLM for multi-hop QA. We introduce, "GenSco", a novel approach of selecting passages based on the predicted decomposition of the multi-hop questions}. The framework consists of two distinct LLMs: (i) Generator LLM, which is used for question decomposition and final answer generation; (ii) an auxiliary open-sourced LLM, used as the scorer, to semantically guide the Generator for passage selection. The generator is invoked only once for the answer generation, resulting in a cost-effective and efficient approach. We evaluate on three broadly established multi-hop question answering datasets: 2WikiMultiHop, Adversarial HotPotQA and MuSiQue and achieve an absolute gain of $15.1$ and $5.9$ points in Exact Match score with respect to the best performing baselines over MuSiQue and 2WikiMultiHop respectively.