Dialogue Model Optimization via Agent Game and Adaptive Tree-based GRPO

TL;DR

This paper introduces a novel RL framework for dialogue agents that combines online personalization with adaptive tree-based policy optimization, enabling long-term, engaging, and personalized conversations with improved efficiency.

Contribution

It proposes a new long-horizon RL method using adaptive tree-based policy optimization and a two-agent game paradigm for better dialogue personalization and efficiency.

Findings

Outperforms existing dialogue models in engagement and personalization.

Achieves higher sample efficiency and robustness in experiments.

Reduces computational overhead from exponential to polynomial in dialogue length.

Abstract

Open-ended dialogue agents aim to deliver engaging, personalized interactions by adapting to users' traits, but existing methods face critical limitations: over-reliance on pre-collected user data, and short-horizon biases in reinforcement learning (RL) that neglect long-term dialogue value. To address these, we propose a novel long-horizon RL framework integrating online personalization with Adaptive Tree-based Group Relative Policy Optimization (AT-GRPO). Adopting a two-agent game paradigm, a user agent constructs dynamic environments via style mimicry (learning user-specific conversational traits) and active termination (predicting turn-level termination probabilities as immediate rewards), forming an iterative cycle that drives the dialogue agent to deepen interest exploration. AT-GRPO reinterprets dialogue trajectories as trees and introduces adaptive observation ranges. Unlike full tree expansion that incurs exponential overhead, it limits each node to aggregate rewards from a stage-aware range: larger ranges support early-stage topic exploration, while smaller ranges facilitate late-stage dialogue maintenance. This design reduces rollout budgets from exponential to polynomial in the dialogue length, while preserving long-term reward capture. Extensive experiments show our framework's superior performance, sample efficiency, and robustness.