Database Search Results Disambiguation for Task-Oriented Dialog Systems

TL;DR

This paper introduces DSR Disambiguation, a new task for resolving multiple database search results in task-oriented dialogs, and provides augmented datasets and methods to improve model handling of ambiguous queries.

Contribution

It proposes a novel disambiguation task, augments existing datasets with synthetic and human paraphrased turns, and demonstrates improved model performance through pre-fine tuning and multi-task learning.

Findings

Training on augmented data improves disambiguation ability.

Pre-fine tuning enhances performance without in-domain data.

Models can learn disambiguation as a universal dialog skill.

Abstract

As task-oriented dialog systems are becoming increasingly popular in our lives, more realistic tasks have been proposed and explored. However, new practical challenges arise. For instance, current dialog systems cannot effectively handle multiple search results when querying a database, due to the lack of such scenarios in existing public datasets. In this paper, we propose Database Search Result (DSR) Disambiguation, a novel task that focuses on disambiguating database search results, which enhances user experience by allowing them to choose from multiple options instead of just one. To study this task, we augment the popular task-oriented dialog datasets (MultiWOZ and SGD) with turns that resolve ambiguities by (a) synthetically generating turns through a pre-defined grammar, and (b) collecting human paraphrases for a subset. We find that training on our augmented dialog data improves the model's ability to deal with ambiguous scenarios, without sacrificing performance on unmodified turns. Furthermore, pre-fine tuning and multi-task learning help our model to improve performance on DSR-disambiguation even in the absence of in-domain data, suggesting that it can be learned as a universal dialog skill. Our data and code will be made publicly available.