LLMs Know What They Need: Leveraging a Missing Information Guided Framework to Empower Retrieval-Augmented Generation

TL;DR

This paper introduces MIGRES, a framework that enables LLMs to identify missing information during reasoning, guiding targeted retrieval and extraction to improve retrieval-augmented generation performance.

Contribution

The paper proposes a novel Missing Information Guided Retrieve-Extraction-Solving paradigm that enhances RAG by leveraging LLMs' ability to detect missing info and guide retrieval and extraction processes.

Findings

MIGRES outperforms existing methods on multiple datasets.

The approach effectively filters irrelevant documents.

Targeted retrieval improves reasoning accuracy.

Abstract

Retrieval-Augmented Generation (RAG) demonstrates great value in alleviating outdated knowledge or hallucination by supplying LLMs with updated and relevant knowledge. However, there are still several difficulties for RAG in understanding complex multi-hop query and retrieving relevant documents, which require LLMs to perform reasoning and retrieve step by step. Inspired by human's reasoning process in which they gradually search for the required information, it is natural to ask whether the LLMs could notice the missing information in each reasoning step. In this work, we first experimentally verified the ability of LLMs to extract information as well as to know the missing. Based on the above discovery, we propose a Missing Information Guided Retrieve-Extraction-Solving paradigm (MIGRES), where we leverage the identification of missing information to generate a targeted query that steers the subsequent knowledge retrieval. Besides, we design a sentence-level re-ranking filtering approach to filter the irrelevant content out from document, along with the information extraction capability of LLMs to extract useful information from cleaned-up documents, which in turn to bolster the overall efficacy of RAG. Extensive experiments conducted on multiple public datasets reveal the superiority of the proposed MIGRES method, and analytical experiments demonstrate the effectiveness of our proposed modules.