Aligning LLMs Toward Multi-Turn Conversational Outcomes Using Iterative PPO

TL;DR

This paper introduces Iterative PPO, a method for training large language models in multi-turn conversations by reducing the problem to single-turn RLHF-style tasks, enabling stable and effective policy improvement.

Contribution

It formalizes a reduction from multi-turn to single-turn RL problems and proposes Iterative PPO, leveraging existing RLHF tools for improved multi-turn conversational AI training.

Findings

Iterative PPO effectively improves multi-turn conversational outcomes.

The method leverages off-the-shelf RLHF tools for stability.

It balances online adaptability with offline training stability.

Abstract

Optimizing large language models (LLMs) for multi-turn conversational outcomes remains a significant challenge, especially in goal-oriented settings like AI marketing or sales agents who facilitate transactions via messaging platforms. The difficulty stems from sparse, long-horizon rewards and the discrepancy between response-level planning and token-level generation. In this technical note, we propose a formal reduction of the multi-turn RL problem into a sequence of single-turn RLHF-style problems. This is achieved by setting a learned multi-turn Q-function as the reward model for the single-turn problem. We demonstrate and prove a key insight: solving this single-turn RL problem with standard token-level PPO is equivalent to a policy improvement step within the multi-turn problem. This insight naturally leads to Iterative PPO, a batch online policy iteration algorithm that alternates between fitting Q-functions from logged conversation trajectories and improving the policy. A major practical advantage is that Iterative PPO directly leverages stable, off-the-shelf single-turn RLHF tools, making it straightforward to implement. Our method occupies a middle ground between fully online and fully offline approaches, retaining the adaptability of online updates while gaining the stability benefits of offline training.