Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Reward Design

TL;DR

This paper introduces turn-level reward design for multi-turn RL in LLM agents, significantly improving reasoning performance, stability, and convergence in complex multi-turn tasks.

Contribution

It systematically studies turn-level reward design and extends RL algorithms like GRPO and PPO for multi-turn reasoning, enabling finer credit assignment and better performance.

Findings

Enhanced accuracy in multi-turn reasoning tasks

Faster convergence and greater stability in training

Achieved 100% format correctness across datasets

Abstract

This paper investigates Reinforcement Learning (RL) approaches to enhance the reasoning capabilities of Large Language Model (LLM) agents in long-horizon, multi-turn scenarios. Although RL algorithms such as Group Relative Policy Optimization (GRPO) and Proximal Policy Optimization (PPO) have been widely applied to train multi-turn LLM agents, they typically rely only on sparse outcome rewards and lack dense intermediate signals across multiple decision steps, limiting their performance on complex reasoning tasks. To bridge this gap, we present the first systematic study of \textit{turn-level reward design} for multi-turn RL algorithms and agent applications. By integrating turn-level rewards, we extend GRPO and PPO to their respective multi-turn variants, enabling fine-grained credit assignment. We conduct case studies on multi-turn reasoning-augmented search agents, where we carefully design two types of turn-level rewards: verifiable and LLM-as-judge. Our experiments on multi-turn search tasks demonstrate that incorporating well-designed turn-level rewards enables RL algorithms to significantly outperform baseline methods with trajectory-level rewards. Both training and validation reward curves illustrate that our method achieves \textit{greater stability}, \textit{faster convergence}, and \textit{higher accuracy}. Numerical results across diverse question-answering datasets further show that our approach consistently delivers highest answer correctness and 100\% format correctness.