RADE: Reference-Assisted Dialogue Evaluation for Open-Domain Dialogue

TL;DR

This paper introduces RADE, a novel reference-assisted evaluation method for open-domain dialogue systems that improves automatic scoring by leveraging reference responses and auxiliary tasks, aligning better with human judgments.

Contribution

The paper presents RADE, a multi-task learning framework that uses reference responses and auxiliary response generation to enhance automatic dialogue evaluation.

Findings

RADE outperforms existing methods in correlation with human judgments.

Extended datasets with human-rated responses support the evaluation.

Experiments show improved consistency with human evaluations.

Abstract

Evaluating open-domain dialogue systems is challenging for reasons such as the one-to-many problem, i.e., many appropriate responses other than just the golden response. As of now, automatic evaluation methods need better consistency with humans, while reliable human evaluation can be time- and cost-intensive. To this end, we propose the Reference-Assisted Dialogue Evaluation (RADE) approach under the multi-task learning framework, which leverages the pre-created utterance as reference other than the gold response to relief the one-to-many problem. Specifically, RADE explicitly compares reference and the candidate response to predict their overall scores. Moreover, an auxiliary response generation task enhances prediction via a shared encoder. To support RADE, we extend three datasets with additional rated responses other than just a golden response by human annotation. Experiments on our three datasets and two existing benchmarks demonstrate the effectiveness of our method, where Pearson, Spearman, and Kendall correlations with human evaluation outperform state-of-the-art baselines.