RAG-R1: Incentivizing the Search and Reasoning Capabilities of LLMs through Multi-query Parallelism

TL;DR

RAG-R1 introduces a multi-query parallelism training framework for LLMs, enhancing reasoning robustness and reducing latency by enabling adaptive knowledge leveraging during inference.

Contribution

The paper presents a novel two-stage training framework that shifts from single-query to multi-query parallelism, improving reasoning and efficiency in LLMs.

Findings

Outperforms baseline by up to 13.7% on QA benchmarks

Reduces inference time by 11.1%

Enhances reasoning robustness with multi-query parallelism

Abstract

Large Language Models (LLMs), despite their remarkable capabilities, are prone to generating hallucinated or outdated content due to their static internal knowledge. While Retrieval-Augmented Generation (RAG) integrated with Reinforcement Learning (RL) offers a solution, these methods are fundamentally constrained by a single-query mode, leading to prohibitive latency and inherent brittleness. To overcome these limitations, we introduce RAG-R1, a novel two-stage training framework centered around multi-query parallelism. Our framework enables LLMs to adaptively leverage internal and external knowledge during the reasoning process while transitioning from the single-query mode to multi-query parallelism. This architectural shift bolsters reasoning robustness while significantly reducing inference latency. Extensive experiments on seven question-answering benchmarks confirm the superiority of our method, which outperforms the strongest baseline by up to 13.7% and decreases inference time by 11.1%.