SmartSearch: Process Reward-Guided Query Refinement for Search Agents

TL;DR

SmartSearch introduces a process reward-guided framework with query refinement and curriculum learning to enhance the quality and efficiency of search queries generated by LLM-based search agents, leading to improved retrieval performance.

Contribution

It proposes a novel framework combining process rewards, query refinement, and curriculum learning to improve intermediate query quality in search agents.

Findings

Outperforms existing baselines in search efficiency.

Achieves significant improvements in query quality.

Demonstrates effectiveness across multiple experimental settings.

Abstract

Large language model (LLM)-based search agents have proven promising for addressing knowledge-intensive problems by incorporating information retrieval capabilities. Existing works largely focus on optimizing the reasoning paradigms of search agents, yet the quality of intermediate search queries during reasoning remains overlooked. As a result, the generated queries often remain inaccurate, leading to unexpected retrieval results and ultimately limiting search agents' overall effectiveness. To mitigate this issue, we introduce SmartSearch, a framework built upon two key mechanisms: (1) Process rewards, which provide fine-grained supervision for the quality of each intermediate search query through Dual-Level Credit Assessment. (2) Query refinement, which promotes the optimization of query generation by selectively refining low-quality search queries and regenerating subsequent search rounds based on these refinements. To enable the search agent to progressively internalize the ability to improve query quality under the guidance of process rewards, we design a three-stage curriculum learning framework. This framework guides the agent through a progression from imitation, to alignment, and ultimately to generalization. Experimental results show that SmartSearch consistently surpasses existing baselines, and additional quantitative analyses further confirm its significant gains in both search efficiency and query quality. The code is available at https://github.com/MYVAE/SmartSearch.