EvolveSearch: An Iterative Self-Evolving Search Agent

TL;DR

EvolveSearch introduces an iterative self-evolving framework combining supervised fine-tuning and reinforcement learning to improve large language model web search capabilities without external data, achieving significant performance gains.

Contribution

It presents a novel self-evolution approach that enhances LLM web search performance through iterative learning, eliminating the need for human-annotated reasoning data.

Findings

Achieves an average of 4.7% improvement over state-of-the-art on seven benchmarks.

Demonstrates consistent performance gains across multiple multi-hop QA tasks.

Shows effectiveness of combining SFT and RL in an iterative framework for web search.

Abstract

The rapid advancement of large language models (LLMs) has transformed the landscape of agentic information seeking capabilities through the integration of tools such as search engines and web browsers. However, current mainstream approaches for enabling LLM web search proficiency face significant challenges: supervised fine-tuning struggles with data production in open-search domains, while RL converges quickly, limiting their data utilization efficiency. To address these issues, we propose EvolveSearch, a novel iterative self-evolution framework that combines SFT and RL to enhance agentic web search capabilities without any external human-annotated reasoning data. Extensive experiments on seven multi-hop question-answering (MHQA) benchmarks demonstrate that EvolveSearch consistently improves performance across iterations, ultimately achieving an average improvement of 4.7\% over the current state-of-the-art across seven benchmarks, opening the door to self-evolution agentic capabilities in open web search domains.