Fantastic Rewards and How to Tame Them: A Case Study on Reward Learning for Task-oriented Dialogue Systems

TL;DR

This paper explores how to efficiently learn and utilize reward functions in training end-to-end task-oriented dialogue agents using reinforcement learning, introducing generalized objectives inspired by learning-to-rank methods, and demonstrating competitive results on Multiwoz 2.0.

Contribution

It proposes novel reward-function learning objectives for dialogue systems and shows how to leverage them to improve training of end-to-end dialogue agents.

Findings

Achieves competitive performance on Multiwoz 2.0 dataset.

Introduces generalized reward learning objectives inspired by learning-to-rank.

Provides publicly available source code and checkpoints.

Abstract

When learning task-oriented dialogue (ToD) agents, reinforcement learning (RL) techniques can naturally be utilized to train dialogue strategies to achieve user-specific goals. Prior works mainly focus on adopting advanced RL techniques to train the ToD agents, while the design of the reward function is not well studied. This paper aims at answering the question of how to efficiently learn and leverage a reward function for training end-to-end (E2E) ToD agents. Specifically, we introduce two generalized objectives for reward-function learning, inspired by the classical learning-to-rank literature. Further, we utilize the learned reward function to guide the training of the E2E ToD agent. With the proposed techniques, we achieve competitive results on the E2E response-generation task on the Multiwoz 2.0 dataset. Source code and checkpoints are publicly released at https://github.com/Shentao-YANG/Fantastic_Reward_ICLR2023.