TSPO: Breaking the Double Homogenization Dilemma in Multi-turn Search Policy Optimization

TL;DR

This paper introduces TSPO, a novel reinforcement learning method that enhances multi-turn search reasoning in LLMs by providing step-level rewards, significantly improving performance over existing approaches.

Contribution

TSPO employs the First-Occurrence Latent Reward mechanism to preserve process signals and boost reward variance without external annotations, addressing the double homogenization dilemma.

Findings

TSPO achieves 24% average performance gain on Qwen2.5-3B models.

TSPO outperforms state-of-the-art baselines in multi-turn search reasoning.

The method enhances intra-group advantage estimation efficiency.

Abstract

Multi-turn tool-integrated reasoning enables Large Language Models (LLMs) to solve complex tasks through iterative information retrieval. However, current reinforcement learning (RL) frameworks for search-augmented reasoning predominantly rely on sparse outcome-level rewards, leading to a "Double Homogenization Dilemma." This manifests as (1) Process homogenization, where the thinking, reasoning, and tooling involved in generation are ignored. (2) Intra-group homogenization, coarse-grained outcome rewards often lead to inefficiencies in intra-group advantage estimation with methods like Group Relative Policy Optimization (GRPO) during sampling. To address this, we propose Turn-level Stage-aware Policy Optimization (TSPO). TSPO introduces the First-Occurrence Latent Reward (FOLR) mechanism, allocating partial rewards to the step where the ground-truth answer first appears, thereby preserving process-level signals and increasing reward variance within groups without requiring external reward models or any annotations. Extensive experiments demonstrate that TSPO significantly outperforms state-of-the-art baselines, achieving average performance gains of 24% and 13.6% on Qwen2.5-3B and 7B models, respectively. Code is available at https://github.com/Flipped-May/TSPO.