Learning End-to-End Goal-Oriented Dialog with Multiple Answers

TL;DR

This paper addresses the challenge of multiple valid responses in goal-oriented dialogs by proposing a new method combining supervised and reinforcement learning, and introduces a more realistic testbed to evaluate dialog systems.

Contribution

It introduces a novel approach that combines supervised and reinforcement learning to handle multiple valid responses and presents a new testbed for realistic evaluation.

Findings

Existing methods' accuracy drops significantly on the new testbed.

The proposed method improves accuracy to 47.3% on permuted-bAbI tasks.

Performance gap highlights the need for better models in realistic dialog settings.

Abstract

In a dialog, there can be multiple valid next utterances at any point. The present end-to-end neural methods for dialog do not take this into account. They learn with the assumption that at any time there is only one correct next utterance. In this work, we focus on this problem in the goal-oriented dialog setting where there are different paths to reach a goal. We propose a new method, that uses a combination of supervised learning and reinforcement learning approaches to address this issue. We also propose a new and more effective testbed, permuted-bAbI dialog tasks, by introducing multiple valid next utterances to the original-bAbI dialog tasks, which allows evaluation of goal-oriented dialog systems in a more realistic setting. We show that there is a significant drop in performance of existing end-to-end neural methods from 81.5% per-dialog accuracy on original-bAbI dialog tasks to 30.3% on permuted-bAbI dialog tasks. We also show that our proposed method improves the performance and achieves 47.3% per-dialog accuracy on permuted-bAbI dialog tasks.