Guided Dialog Policy Learning: Reward Estimation for Multi-Domain Task-Oriented Dialog

TL;DR

This paper introduces Guided Dialog Policy Learning, an adversarial inverse reinforcement learning approach that jointly estimates rewards and optimizes policies for multi-domain task-oriented dialogs, improving success rates.

Contribution

It presents a novel reward estimation method that handles complex multi-domain goals without manual reward design, enhancing dialog policy learning.

Findings

Achieves higher task success rates than baselines

Effectively infers user goals during dialogs

Improves policy guidance through learned rewards

Abstract

Dialog policy decides what and how a task-oriented dialog system will respond, and plays a vital role in delivering effective conversations. Many studies apply Reinforcement Learning to learn a dialog policy with the reward function which requires elaborate design and pre-specified user goals. With the growing needs to handle complex goals across multiple domains, such manually designed reward functions are not affordable to deal with the complexity of real-world tasks. To this end, we propose Guided Dialog Policy Learning, a novel algorithm based on Adversarial Inverse Reinforcement Learning for joint reward estimation and policy optimization in multi-domain task-oriented dialog. The proposed approach estimates the reward signal and infers the user goal in the dialog sessions. The reward estimator evaluates the state-action pairs so that it can guide the dialog policy at each dialog turn. Extensive experiments on a multi-domain dialog dataset show that the dialog policy guided by the learned reward function achieves remarkably higher task success than state-of-the-art baselines.