AT$^2$PO: Agentic Turn-based Policy Optimization via Tree Search

TL;DR

This paper introduces AT$^2$PO, a novel framework for multi-turn agentic reinforcement learning that combines tree search with turn-level policy optimization to improve exploration, credit assignment, and policy alignment.

Contribution

AT$^2$PO presents a unified turn-level tree structure and learning objective, enhancing multi-turn RL by addressing exploration, credit assignment, and policy optimization challenges.

Findings

Achieves up to 1.84% improvement over state-of-the-art baselines.

Validates effectiveness through extensive ablation studies.

Demonstrates versatility by integrating with various RL pipelines.

Abstract

LLM agents have emerged as powerful systems for tackling multi-turn tasks by interleaving internal reasoning and external tool interactions. Agentic Reinforcement Learning has recently drawn significant research attention as a critical post-training paradigm to further refine these capabilities. In this paper, we present AT$^2$PO (Agentic Turn-based Policy Optimization via Tree Search), a unified framework for multi-turn agentic RL that addresses three core challenges: limited exploration diversity, sparse credit assignment, and misaligned policy optimization. AT$^2$PO introduces a turn-level tree structure that jointly enables Entropy-Guided Tree Expansion for strategic exploration and Turn-wise Credit Assignment for fine-grained reward propagation from sparse outcomes. Complementing this, we propose Agentic Turn-based Policy Optimization, a turn-level learning objective that aligns policy updates with the natural decision granularity of agentic interactions. ATPO is orthogonal to tree search and can be readily integrated into any multi-turn RL pipeline. Experiments across seven benchmarks demonstrate consistent improvements over the state-of-the-art baseline by up to 1.84 percentage points in average, with ablation studies validating the effectiveness of each component. Our code is available at https://github.com/zzfoutofspace/ATPO.