Implicit Turn-Wise Policy Optimization for Proactive User-LLM Interaction

TL;DR

This paper introduces Implicit Turn-wise Policy Optimization (ITPO), a reinforcement learning method that derives fine-grained, turn-level rewards from sparse signals to improve multi-turn human-AI interactions in various collaborative tasks.

Contribution

ITPO is a novel reinforcement learning approach that uses an implicit process reward model to enhance training stability and effectiveness in multi-turn AI-human collaboration.

Findings

ITPO improves convergence across multiple tasks.

Turn-wise rewards align with human judgment.

Enhanced training stability with normalization.

Abstract

Multi-turn human-AI collaboration is fundamental to deploying interactive services such as adaptive tutoring, conversational recommendation, and professional consultation. However, optimizing these interactions via reinforcement learning is hindered by the sparsity of verifiable intermediate rewards and the high stochasticity of user responses. To address these challenges, we introduce Implicit Turn-wise Policy Optimization (ITPO). ITPO leverages an implicit process reward model to derive fine-grained, turn-wise process rewards from sparse outcome signals. Unlike volatile token-level rewards, these turn-level signals exhibit superior robustness and may utilize a normalization mechanism to further enhance training stability. We evaluate ITPO across three representative multi-turn collaborative tasks: math tutoring, document writing, and medical recommendation. Empirical results demonstrate that ITPO, when combined with PPO, GRPO, or RLOO, consistently achieves improved convergence than existing baselines. Elaborate trajectory analysis confirms that ITPO infers turn-wise preferences that are semantically aligned with human judgment. Code is publicly available at https://github.com/Graph-COM/ITPO.