IM-RAG: Multi-Round Retrieval-Augmented Generation Through Learning Inner Monologues

TL;DR

IM-RAG introduces a novel multi-round retrieval-augmented generation framework that leverages learned inner monologues and reinforcement learning to improve reasoning, interpretability, and flexibility in large language models.

Contribution

The paper presents IM-RAG, a new LLM-centric approach integrating IR systems with learned inner monologues, optimized via RL and SFT for enhanced multi-round reasoning.

Findings

Achieves state-of-the-art results on HotPotQA.

Provides high interpretability through learned inner monologues.

Enhances flexibility in IR module integration.

Abstract

Although the Retrieval-Augmented Generation (RAG) paradigms can use external knowledge to enhance and ground the outputs of Large Language Models (LLMs) to mitigate generative hallucinations and static knowledge base problems, they still suffer from limited flexibility in adopting Information Retrieval (IR) systems with varying capabilities, constrained interpretability during the multi-round retrieval process, and a lack of end-to-end optimization. To address these challenges, we propose a novel LLM-centric approach, IM-RAG, that integrates IR systems with LLMs to support multi-round RAG through learning Inner Monologues (IM, i.e., the human inner voice that narrates one's thoughts). During the IM process, the LLM serves as the core reasoning model (i.e., Reasoner) to either propose queries to collect more information via the Retriever or to provide a final answer based on the conversational context. We also introduce a Refiner that improves the outputs from the Retriever, effectively bridging the gap between the Reasoner and IR modules with varying capabilities and fostering multi-round communications. The entire IM process is optimized via Reinforcement Learning (RL) where a Progress Tracker is incorporated to provide mid-step rewards, and the answer prediction is further separately optimized via Supervised Fine-Tuning (SFT). We conduct extensive experiments with the HotPotQA dataset, a popular benchmark for retrieval-based, multi-step question-answering. The results show that our approach achieves state-of-the-art (SOTA) performance while providing high flexibility in integrating IR modules as well as strong interpretability exhibited in the learned inner monologues.