Designing Precise and Robust Dialogue Response Evaluators

TL;DR

This paper introduces a new reference-free dialogue response evaluator leveraging semi-supervised training and pretrained language models, achieving high correlation with human judgment and robustness across diverse datasets.

Contribution

It presents a novel reference-free evaluation method that outperforms existing metrics in correlation and robustness, using semi-supervised learning and pretrained models.

Findings

Achieves correlation > 0.6 with human judgment

Generalizes well across diverse responses and datasets

Open-sourced code and data for reproducibility

Abstract

Automatic dialogue response evaluator has been proposed as an alternative to automated metrics and human evaluation. However, existing automatic evaluators achieve only moderate correlation with human judgement and they are not robust. In this work, we propose to build a reference-free evaluator and exploit the power of semi-supervised training and pretrained (masked) language models. Experimental results demonstrate that the proposed evaluator achieves a strong correlation (> 0.6) with human judgement and generalizes robustly to diverse responses and corpora. We open-source the code and data in https://github.com/ZHAOTING/dialog-processing.