Don't Forget Your ABC's: Evaluating the State-of-the-Art in Chat-Oriented Dialogue Systems

TL;DR

This paper introduces a new human evaluation method for open-domain chat systems that provides more reliable, multi-dimensional assessments compared to traditional approaches, addressing high variance and standardization issues.

Contribution

The paper proposes a novel behavior-based human evaluation method for dialogue systems and demonstrates its effectiveness over existing Likert-style and comparative approaches.

Findings

The new method offers more consistent and reliable evaluations.

It effectively captures multiple dialogue behaviors.

It outperforms existing evaluation approaches in reliability.

Abstract

Despite tremendous advancements in dialogue systems, stable evaluation still requires human judgments producing notoriously high-variance metrics due to their inherent subjectivity. Moreover, methods and labels in dialogue evaluation are not fully standardized, especially for open-domain chats, with a lack of work to compare and assess the validity of those approaches. The use of inconsistent evaluation can misinform the performance of a dialogue system, which becomes a major hurdle to enhance it. Thus, a dimensional evaluation of chat-oriented open-domain dialogue systems that reliably measures several aspects of dialogue capabilities is desired. This paper presents a novel human evaluation method to estimate the rates of many dialogue system behaviors. Our method is used to evaluate four state-of-the-art open-domain dialogue systems and compared with existing approaches. The analysis demonstrates that our behavior method is more suitable than alternative Likert-style or comparative approaches for dimensional evaluation of these systems.