Question Decomposition for Retrieval-Augmented Generation

TL;DR

This paper introduces a question decomposition and reranking approach for retrieval-augmented generation to improve multi-hop question answering by assembling and selecting the most relevant evidence from multiple documents.

Contribution

It proposes a practical RAG pipeline that combines LLM-based question decomposition with reranking, enhancing multi-hop retrieval without additional training or specialized indexing.

Findings

Significant improvements in retrieval accuracy (MRR@10 +36.7%)

Enhanced answer accuracy (F1 +11.6%)

Effective assembly of complementary documents for multi-hop questions

Abstract

Grounding large language models (LLMs) in verifiable external sources is a well-established strategy for generating reliable answers. Retrieval-augmented generation (RAG) is one such approach, particularly effective for tasks like question answering: it retrieves passages that are semantically related to the question and then conditions the model on this evidence. However, multi-hop questions, such as "Which company among NVIDIA, Apple, and Google made the biggest profit in 2023?," challenge RAG because relevant facts are often distributed across multiple documents rather than co-occurring in one source, making it difficult for standard RAG to retrieve sufficient information. To address this, we propose a RAG pipeline that incorporates question decomposition: (i) an LLM decomposes the original query into sub-questions, (ii) passages are retrieved for each sub-question, and (iii) the merged candidate pool is reranked to improve the coverage and precision of the retrieved evidence. We show that question decomposition effectively assembles complementary documents, while reranking reduces noise and promotes the most relevant passages before answer generation. Although reranking itself is standard, we show that pairing an off-the-shelf cross-encoder reranker with LLM-driven question decomposition bridges the retrieval gap on multi-hop questions and provides a practical, drop-in enhancement, without any extra training or specialized indexing. We evaluate our approach on the MultiHop-RAG and HotpotQA, showing gains in retrieval (MRR@10: +36.7%) and answer accuracy (F1: +11.6%) over standard RAG baselines.