RAG-Instruct: Boosting LLMs with Diverse Retrieval-Augmented Instructions

TL;DR

RAG-Instruct introduces a versatile method to generate diverse, high-quality retrieval-augmented instructions from any source, significantly improving LLMs' ability to handle various RAG scenarios and tasks.

Contribution

It presents a general approach to synthesize a large, diverse RAG instruction dataset from Wikipedia, enhancing LLMs' retrieval-augmented capabilities across multiple scenarios.

Findings

Constructed a 40K instruction dataset covering diverse RAG tasks

Achieved strong zero-shot performance on RAG benchmarks

Outperformed existing RAG baselines significantly

Abstract

Retrieval-Augmented Generation (RAG) has emerged as a key paradigm for enhancing large language models (LLMs) by incorporating external knowledge. However, current RAG methods face two limitations: (1) they only cover limited RAG scenarios. (2) They suffer from limited task diversity due to the lack of a general RAG dataset. To address these limitations, we propose RAG-Instruct, a general method for synthesizing diverse and high-quality RAG instruction data based on any source corpus. Our approach leverages (1) five RAG paradigms, which encompass diverse query-document relationships, and (2) instruction simulation, which enhances instruction diversity and quality by utilizing the strengths of existing instruction datasets. Using this method, we construct a 40K instruction dataset from Wikipedia, comprehensively covering diverse RAG scenarios and tasks. Experiments demonstrate that RAG-Instruct effectively enhances LLMs' RAG capabilities, achieving strong zero-shot performance and significantly outperforming various RAG baselines across a diverse set of tasks. RAG-Instruct is publicly available at https://github.com/FreedomIntelligence/RAG-Instruct.