Reinforcement Fine-Tuning for Reasoning towards Multi-Step Multi-Source Search in Large Language Models

TL;DR

This paper introduces R-Search, a unified single-LLM framework for multi-step, multi-source search and reasoning, utilizing reinforcement fine-tuning to improve factual accuracy and efficiency in time-sensitive queries.

Contribution

It proposes a novel structured output format and a reinforcement fine-tuning method, ReFT, enabling an LLM to perform integrated multi-source search and reasoning within one inference process.

Findings

Outperforms state-of-the-art methods on multiple benchmarks.

Achieves 70% reduction in context token usage.

Reduces execution latency by approximately 50%.

Abstract

Large language models (LLMs) can face factual limitations when responding to time-sensitive queries about recent events that arise after their knowledge thresholds in the training corpus. Existing search-augmented approaches fall into two categories, each with distinct limitations: multi-agent search frameworks incur substantial computational overhead by separating search planning and response synthesis across multiple LLMs, while single-LLM tool-calling methods restrict themselves to sequential planned, single-query searches from sole search sources. We present Reasoning-Search (R-Search), a single-LLM search framework that unifies multi-step planning, multi-source search execution, and answer synthesis within one coherent inference process. Innovatively, it structure the output into four explicitly defined components, including reasoning steps that guide the search process (<think>), a natural-language directed acyclic graph that represents the search plans with respect to diverse sources (<search>), retrieved results from executing the search plans (<result>), and synthesized final answers (<answer>). To enable effective generation of these structured outputs, we propose a specialized Reinforcement Fine-Tuning (ReFT) method based on GRPO, together with a multi-component reward function that optimizes LLM's answer correctness, structural validity of the generated DAG, and adherence to the defined output format. Experimental evaluation on FinSearchBench-24, SearchExpertBench-25, and seven Q and A benchmarks demonstrates that R-Search outperforms state-of-the-art methods, while achieving substantial efficiency gains through 70% reduction in context token usage and approximately 50% decrease in execution latency. Code is available at https://github.com/wentao0429/Reasoning-search.