Atom-Searcher: Enhancing Agentic Deep Research via Fine-Grained Atomic Thought Reward

TL;DR

Atom-Searcher introduces a fine-grained reasoning framework with Atomic Thought and reward models, significantly improving multi-hop reasoning and strategic search in LLMs through a curriculum-based RL approach.

Contribution

The paper proposes Atomic Thought and ATR to enhance RL-based agentic research, enabling better reasoning, interpretability, and scalability in LLMs.

Findings

Consistent performance improvements on seven benchmarks.

Enhanced interpretability and human-like reasoning patterns.

Scalable computation at test-time.

Abstract

Large language models (LLMs) exhibit remarkable problem-solving abilities, but struggle with complex tasks due to static internal knowledge. Retrieval-Augmented Generation (RAG) enhances access to external information, yet remains limited in multi-hop reasoning and strategic search due to rigid workflows. Recent advancements in agentic deep research empower LLMs to autonomously reason, search, and synthesize information. However, current approaches relying on outcome-based reinforcement learning (RL) face critical issues such as conflicting gradients and reward sparsity, limiting performance gains and training efficiency. To address these, we first propose Atomic Thought, a novel LLM thinking paradigm that decomposes reasoning into fine-grained functional units. These units are supervised by Reasoning Reward Models (RRMs), which provide Atomic Thought Rewards (ATR) for fine-grained guidance. Building on this, we propose Atom-Searcher, a novel RL framework for agentic deep research that integrates Atomic Thought and ATR. Atom-Searcher uses a curriculum-inspired reward schedule, prioritizing process-level ATR early and transitioning to outcome rewards, accelerating convergence on effective reasoning paths. Experiments on seven benchmarks show consistent improvements over the state-of-the-art. Key advantages include: (1) Atom-Searcher scales computation at test-time. (2) Atomic Thought provides supervision anchors for RRMs, bridging deep research tasks and RRMs. (3) Atom-Searcher exhibits more interpretable, human-like reasoning patterns.