RetroLLM: Empowering Large Language Models to Retrieve Fine-grained Evidence within Generation

TL;DR

RetroLLM is a unified framework that enhances large language models by enabling them to retrieve and generate fine-grained evidence directly from a corpus, reducing hallucinations and improving accuracy in open-domain QA tasks.

Contribution

It introduces a joint retrieval-generation approach with hierarchical constraints and forward-looking decoding, addressing limitations of existing retrieval-augmented methods.

Findings

Outperforms existing methods on five open-domain QA datasets

Reduces irrelevant decoding through hierarchical FM-Index constraints

Improves evidence accuracy with forward-looking constrained decoding

Abstract

Large language models (LLMs) exhibit remarkable generative capabilities but often suffer from hallucinations. Retrieval-augmented generation (RAG) offers an effective solution by incorporating external knowledge, but existing methods still face several limitations: additional deployment costs of separate retrievers, redundant input tokens from retrieved text chunks, and the lack of joint optimization of retrieval and generation. To address these issues, we propose \textbf{RetroLLM}, a unified framework that integrates retrieval and generation into a single, cohesive process, enabling LLMs to directly generate fine-grained evidence from the corpus with constrained decoding. Moreover, to mitigate false pruning in the process of constrained evidence generation, we introduce (1) hierarchical FM-Index constraints, which generate corpus-constrained clues to identify a subset of relevant documents before evidence generation, reducing irrelevant decoding space; and (2) a forward-looking constrained decoding strategy, which considers the relevance of future sequences to improve evidence accuracy. Extensive experiments on five open-domain QA datasets demonstrate RetroLLM's superior performance across both in-domain and out-of-domain tasks. The code is available at \url{https://github.com/sunnynexus/RetroLLM}.