CausalScore: An Automatic Reference-Free Metric for Assessing Response Relevance in Open-Domain Dialogue Systems

TL;DR

CausalScore is a novel reference-free metric that evaluates response relevance in open-domain dialogue systems by measuring causal strength, significantly improving alignment with human judgments over existing metrics.

Contribution

We introduce CausalScore, a new causal-based metric for response evaluation, and provide a new dataset with human-annotated causal relations to support future research.

Findings

CausalScore outperforms existing metrics in aligning with human judgments.

It effectively measures relevance through causal strength between dialogue history and responses.

The new dataset CGDIALOG+ facilitates future development of dialogue evaluation metrics.

Abstract

Automatically evaluating the quality of responses in open-domain dialogue systems is a challenging but crucial task. Current evaluation metrics often fail to align with human judgments, especially when assessing responses that are grammatically correct. To address this issue, we propose a novel metric, called CausalScore, which assesses the relevance of responses by measuring the causal strength between dialogue histories and responses. The causal strength is estimated by utilizing both unconditional dependence and conditional dependencies from the dialogue history to responses. We compare our metric with the existing competitive metrics in terms of their alignment with human judgements. Our experimental results demonstrate that CausalScore significantly surpasses existing state-of-the-art metrics by aligning better with human judgements. Additionally, we collect a new dialogue dataset CGDIALOG+ with human-annotated causal relations and a set of pairwise human judgements to facilitate the development of future automatic metrics.