Oh My Mistake!: Toward Realistic Dialogue State Tracking including Turnback Utterances

TL;DR

This paper highlights that current dialogue state tracking benchmarks lack realistic turnback utterances, causing performance drops, but these issues can be mitigated by incorporating such scenarios into training datasets.

Contribution

The study reveals the limitations of existing DST benchmarks in handling turnback utterances and demonstrates that explicit training on turnback scenarios improves model robustness.

Findings

DST models perform poorly with turnback utterances in current benchmarks.

Including turnback scenarios in training data improves DST performance.

Benchmark datasets need to incorporate realistic turnback utterances for better evaluation.

Abstract

The primary purpose of dialogue state tracking (DST), a critical component of an end-to-end conversational system, is to build a model that responds well to real-world situations. Although we often change our minds from time to time during ordinary conversations, current benchmark datasets do not adequately reflect such occurrences and instead consist of over-simplified conversations, in which no one changes their mind during a conversation. As the main question inspiring the present study, "Are current benchmark datasets sufficiently diverse to handle casual conversations in which one changes their mind after a certain topic is over?" We found that the answer is "No" because DST models cannot refer to previous user preferences when template-based turnback utterances are injected into the dataset. Even in the the simplest mind-changing (turnback) scenario, the performance of DST models significantly degenerated. However, we found that this performance degeneration can be recovered when the turnback scenarios are explicitly designed in the training set, implying that the problem is not with the DST models but rather with the construction of the benchmark dataset.