PaperScout: An Autonomous Agent for Academic Paper Search with Process-Aware Sequence-Level Policy Optimization

TL;DR

PaperScout introduces an autonomous, process-aware agent for academic paper search that dynamically manages search actions, optimized with a novel sequence-level policy method, significantly improving retrieval performance over traditional workflows.

Contribution

The paper presents PaperScout, a novel autonomous agent framework for paper search, and Proximal Sequence Policy Optimization (PSPO), a new training method for multi-turn decision-making tasks.

Findings

Outperforms existing workflow-based and RL baselines in recall and relevance

Demonstrates effectiveness on synthetic and real-world benchmarks

Validates the benefits of adaptive, process-aware search strategies

Abstract

Academic paper search is a fundamental task in scientific research, yet most existing approaches rely on rigid, predefined workflows that struggle with complex, conditional queries. To address this limitation, we propose PaperScout, an autonomous agent that reformulates paper search as a sequential decision-making process. Unlike static workflows, PaperScout dynamically decides whether, when, and how to invoke search and expand tools based on accumulated retrieval context. However, training such agents presents a fundamental challenge: standard reinforcement learning methods, typically designed for single-turn tasks, suffer from a granularity mismatch when applied to multi-turn agentic tasks-where token-level optimization diverges from the granularity of sequence-level interactions-leading to noisy credit assignment and unstable training dynamics. We introduce Proximal Sequence Policy Optimization (PSPO), a process-aware, sequence-level policy optimization method that aligns optimization with agent--environment interaction. Comprehensive experiments on both synthetic and real-world benchmarks demonstrate that PaperScout significantly outperforms strong workflow-driven and RL baselines in both recall and relevance, validating the effectiveness of our adaptive agentic framework and optimization strategy.