Learning from Real Users: Rating Dialogue Success with Neural Networks for Reinforcement Learning in Spoken Dialogue Systems

TL;DR

This paper introduces neural network models that assess dialogue success in spoken dialogue systems using only turn-level features, enabling learning from real users without prior task knowledge.

Contribution

The paper presents novel neural network models that evaluate dialogue success solely from turn features, facilitating learning from real user interactions without prior task information.

Findings

Models trained on simulated data generalize to real user dialogues.

The best model achieves success rate comparable to systems with prior task knowledge.

Neural models can effectively evaluate dialogue success in real-time.

Abstract

To train a statistical spoken dialogue system (SDS) it is essential that an accurate method for measuring task success is available. To date training has relied on presenting a task to either simulated or paid users and inferring the dialogue's success by observing whether this presented task was achieved or not. Our aim however is to be able to learn from real users acting under their own volition, in which case it is non-trivial to rate the success as any prior knowledge of the task is simply unavailable. User feedback may be utilised but has been found to be inconsistent. Hence, here we present two neural network models that evaluate a sequence of turn-level features to rate the success of a dialogue. Importantly these models make no use of any prior knowledge of the user's task. The models are trained on dialogues generated by a simulated user and the best model is then used to train a policy on-line which is shown to perform at least as well as a baseline system using prior knowledge of the user's task. We note that the models should also be of interest for evaluating SDS and for monitoring a dialogue in rule-based SDS.