SE-Search: Self-Evolving Search Agent via Memory and Dense Reward

TL;DR

SE-Search introduces a novel self-evolving search agent that enhances online search and question answering by using memory purification, atomic query training, and dense rewards, leading to significant performance improvements.

Contribution

It presents a new self-evolving search framework with three key components that improve search relevance and efficiency over existing methods.

Findings

Outperforms strong baselines on question answering benchmarks.

Achieves a 10.8 point absolute improvement over Search-R1.

Demonstrates the effectiveness of dense rewards in training speed.

Abstract

Retrieval augmented generation (RAG) reduces hallucinations and factual errors in large language models (LLMs) by conditioning generation on retrieved external knowledge. Recent search agents further cast RAG as an autonomous, multi-turn information-seeking process. However, existing methods often accumulate irrelevant or noisy documents and rely on sparse reinforcement learning signals. We propose \textbf{S}elf-\textbf{E}volving \textbf{Search}, a Self-Evolving Search agent that improves online search behavior through three components, memory purification, atomic query training, and dense rewards. SE-Search follows a \textit{Think-Search-Memorize} strategy that retains salient evidence while filtering irrelevant content. Atomic query training promotes shorter and more diverse queries, improving evidence acquisition. Dense rewards provide fine-grained feedback that speeds training. Experiments on single-hop and multi-hop question answering benchmarks show that \texttt{SE-Search-3B} outperforms strong baselines, yielding a $10.8$ point absolute improvement and a $33.8\%$ relative gain over Search-R1.\footnote{We will make the code and model weights publicly available upon acceptance.}