RVR: Retrieve-Verify-Retrieve for Comprehensive Question Answering

TL;DR

The paper introduces RVR, a multi-round retrieval framework that iteratively enhances answer coverage for complex questions by combining retrieval and verification steps, outperforming existing methods.

Contribution

It proposes a novel retrieve-verify-retrieve framework that improves comprehensive answer retrieval through iterative refinement and retriever fine-tuning.

Findings

Achieves at least 10% relative and 3% absolute gain in complete recall on QAMPARI.

Demonstrates consistent improvements on QUEST and WebQuestionsSP datasets.

Effective even with off-the-shelf retrievers, with further gains from fine-tuning.

Abstract

Comprehensively retrieving diverse documents is crucial to address queries that admit a wide range of valid answers. We introduce retrieve-verify-retrieve (RVR), a multi-round retrieval framework designed to maximize answer coverage. Initially, a retriever takes the original query and returns a candidate document set, followed by a verifier that identifies a high-quality subset. For subsequent rounds, the query is augmented with previously verified documents to uncover answers that are not yet covered in previous rounds. RVR is effective even with off-the-shelf retrievers, and fine-tuning retrievers for our inference procedure brings further gains. Our method outperforms baselines, including agentic search approaches, achieving at least 10% relative and 3% absolute gain in complete recall percentage on a multi-answer retrieval dataset (QAMPARI). We also see consistent gains on two out-of-domain datasets (QUEST and WebQuestionsSP) across different base retrievers. Our work presents a promising iterative approach for comprehensive answer recall leveraging a verifier and adapting retrievers to a new inference scenario.