Dialogue Evaluation with Offline Reinforcement Learning

TL;DR

This paper introduces an offline reinforcement learning-based critic for evaluating task-oriented dialogue systems, which correlates well with human judgments and enables fair comparison across different systems using static corpora.

Contribution

It presents a novel offline RL critic for dialogue evaluation that is corpus- and model-independent, improving correlation with human judgments.

Findings

Offline RL critics correlate strongly with human judgments.

The method enables comparison across various dialogue systems.

The approach is corpus- and model-independent.

Abstract

Task-oriented dialogue systems aim to fulfill user goals through natural language interactions. They are ideally evaluated with human users, which however is unattainable to do at every iteration of the development phase. Simulated users could be an alternative, however their development is nontrivial. Therefore, researchers resort to offline metrics on existing human-human corpora, which are more practical and easily reproducible. They are unfortunately limited in reflecting real performance of dialogue systems. BLEU for instance is poorly correlated with human judgment, and existing corpus-based metrics such as success rate overlook dialogue context mismatches. There is still a need for a reliable metric for task-oriented systems with good generalization and strong correlation with human judgements. In this paper, we propose the use of offline reinforcement learning for dialogue evaluation based on a static corpus. Such an evaluator is typically called a critic and utilized for policy optimization. We go one step further and show that offline RL critics can be trained on a static corpus for any dialogue system as external evaluators, allowing dialogue performance comparisons across various types of systems. This approach has the benefit of being corpus- and model-independent, while attaining strong correlation with human judgements, which we confirm via an interactive user trial.