Simulation-Free Hierarchical Latent Policy Planning for Proactive Dialogues

TL;DR

This paper introduces LDPP, a novel framework for proactive dialogue policy planning that automatically discovers and learns policies from real dialogue data, outperforming existing methods and large language models.

Contribution

The paper presents a fully automated, data-driven approach for discovering and learning dialogue policies using latent space representations and hierarchical reinforcement learning.

Findings

LDPP outperforms existing methods on proactive dialogue scenarios.

LDPP surpasses ChatGPT with a smaller 1.8-billion-parameter model.

The approach effectively automates policy discovery from real dialogue records.

Abstract

Recent advancements in proactive dialogues have garnered significant attention, particularly for more complex objectives (e.g. emotion support and persuasion). Unlike traditional task-oriented dialogues, proactive dialogues demand advanced policy planning and adaptability, requiring rich scenarios and comprehensive policy repositories to develop such systems. However, existing approaches tend to rely on Large Language Models (LLMs) for user simulation and online learning, leading to biases that diverge from realistic scenarios and result in suboptimal efficiency. Moreover, these methods depend on manually defined, context-independent, coarse-grained policies, which not only incur high expert costs but also raise concerns regarding their completeness. In our work, we highlight the potential for automatically discovering policies directly from raw, real-world dialogue records. To this end, we introduce a novel dialogue policy planning framework, LDPP. It fully automates the process from mining policies in dialogue records to learning policy planning. Specifically, we employ a variant of the Variational Autoencoder to discover fine-grained policies represented as latent vectors. After automatically annotating the data with these latent policy labels, we propose an Offline Hierarchical Reinforcement Learning (RL) algorithm in the latent space to develop effective policy planning capabilities. Our experiments demonstrate that LDPP outperforms existing methods on two proactive scenarios, even surpassing ChatGPT with only a 1.8-billion-parameter LLM.