Decoupling Strategy and Execution in Task-Focused Dialogue via Goal-Oriented Preference Optimization

TL;DR

This paper introduces GOPO, a hierarchical reinforcement learning framework that decouples strategy planning from response generation in task-oriented dialogue systems, leading to significant improvements in long-horizon task success and dialogue quality.

Contribution

The paper proposes GOPO, a novel hierarchical RL approach that separates goal optimization from response generation, improving long-term task success in dialogue systems.

Findings

GOPO improves TSE by 7.7% and 10.3% over PPO and Memento on Mgshop.

A 14B model trained with GOPO outperforms Qwen-235B and GPT-5.2 in TSE.

Ablation studies highlight the importance of the Expert Agent in long-horizon optimization.

Abstract

Large language models show potential in task-oriented dialogue systems, yet existing training methods often rely on token-level likelihood or preference optimization, which poorly align with long-horizon task success. To address this, we propose Goal-Oriented Preference Optimization (GOPO), a hierarchical reinforcement learning framework that decouples strategy planning from response generation via an Expert Agent and a Customer Service Agent. The Expert Agent optimizes multi-turn goal preferences at the dialogue-trajectory level, while the Customer Service Agent generates responses strictly aligned with the selected strategy. We evaluate GOPO on public benchmarks and e-commerce customer service datasets, and introduce Task-focused Sequential Engagement (TSE), a sequence-level metric derived from real e-commerce interaction data. On the Mgshop dataset, GOPO improves TSE by 7.7% and 10.3% over PPO and Memento, with consistent gains in sequence-level reward and generation quality. Furthermore, a 14B model trained with GOPO achieves 2.7% and 1.5% higher TSE than Qwen-235B and GPT-5.2, respectively. Ablation studies confirm the Expert Agent's critical role in long-horizon optimization. GOPO demonstrates consistent improvements across other datasets as well. This work establishes a new paradigm for task-oriented dialogue systems in commercial scenarios, with code and datasets to be made public.