Towards a Metric for Automated Conversational Dialogue System Evaluation and Improvement

TL;DR

AutoJudge is an automated evaluation method for conversational dialogue systems that uses self-talk generated dialogues and human ratings to train a judgement model, showing good correlation with human assessments and potential for system improvement.

Contribution

The paper introduces AutoJudge, a novel automated evaluation metric for dialogue systems that leverages self-talk and human ratings, and explores its application for system re-ranking and reinforcement learning.

Findings

AutoJudge correlates well with human ratings.

AutoJudge effectively re-ranks candidate utterances.

AutoJudge cannot currently be used as a reward in reinforcement learning.

Abstract

We present "AutoJudge", an automated evaluation method for conversational dialogue systems. The method works by first generating dialogues based on self-talk, i.e. dialogue systems talking to itself. Then, it uses human ratings on these dialogues to train an automated judgement model. Our experiments show that AutoJudge correlates well with the human ratings and can be used to automatically evaluate dialogue systems, even in deployed systems. In a second part, we attempt to apply AutoJudge to improve existing systems. This works well for re-ranking a set of candidate utterances. However, our experiments show that AutoJudge cannot be applied as reward for reinforcement learning, although the metric can distinguish good from bad dialogues. We discuss potential reasons, but state here already that this is still an open question for further research.