Why Guided Dialog Policy Learning performs well? Understanding the role of adversarial learning and its alternative

TL;DR

This paper analyzes the role of adversarial learning in dialog policy learning and proposes an alternative method that retains benefits without using adversarial learning, demonstrated on a multi-domain dialog dataset.

Contribution

It provides a detailed analysis of adversarial learning's role in dialog policy learning and introduces a new approach that removes AL while maintaining performance.

Findings

Adversarial learning enhances dialog policy performance but has issues like mode collapse.

The proposed method achieves comparable results without adversarial learning.

Evaluation on MultiWOZ shows the effectiveness of the new approach.

Abstract

Dialog policies, which determine a system's action based on the current state at each dialog turn, are crucial to the success of the dialog. In recent years, reinforcement learning (RL) has emerged as a promising option for dialog policy learning (DPL). In RL-based DPL, dialog policies are updated according to rewards. The manual construction of fine-grained rewards, such as state-action-based ones, to effectively guide the dialog policy is challenging in multi-domain task-oriented dialog scenarios with numerous state-action pair combinations. One way to estimate rewards from collected data is to train the reward estimator and dialog policy simultaneously using adversarial learning (AL). Although this method has demonstrated superior performance experimentally, it is fraught with the inherent problems of AL, such as mode collapse. This paper first identifies the role of AL in DPL through detailed analyses of the objective functions of dialog policy and reward estimator. Next, based on these analyses, we propose a method that eliminates AL from reward estimation and DPL while retaining its advantages. We evaluate our method using MultiWOZ, a multi-domain task-oriented dialog corpus.