R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning

TL;DR

R-Search introduces a reinforcement learning framework that enhances large language models' reasoning by optimizing their interaction with search, leading to significant improvements in complex, knowledge-intensive tasks.

Contribution

It presents a novel multi-reward RL approach for dynamic reasoning-search integration, enabling LLMs to better decide when to retrieve information or reason, improving response quality.

Findings

Outperforms RAG baselines by up to 32.2% in-domain

Achieves 25.1% improvement out-of-domain

Effectively learns optimal reasoning-search trajectories

Abstract

Large language models (LLMs) have notably progressed in multi-step and long-chain reasoning. However, extending their reasoning capabilities to encompass deep interactions with search remains a non-trivial challenge, as models often fail to identify optimal reasoning-search interaction trajectories, resulting in suboptimal responses. We propose R-Search, a novel reinforcement learning framework for Reasoning-Search integration, designed to enable LLMs to autonomously execute multi-step reasoning with deep search interaction, and learn optimal reasoning search interaction trajectories via multi-reward signals, improving response quality in complex logic- and knowledge-intensive tasks. R-Search guides the LLM to dynamically decide when to retrieve or reason, while globally integrating key evidence to enhance deep knowledge interaction between reasoning and search. During RL training, R-Search provides multi-stage, multi-type rewards to jointly optimize the reasoning-search trajectory. Experiments on seven datasets show that R-Search outperforms advanced RAG baselines by up to 32.2% (in-domain) and 25.1% (out-of-domain). The code and data are available at https://github.com/QingFei1/R-Search.