FlowEval: A Consensus-Based Dialogue Evaluation Framework Using Segment Act Flows

TL;DR

FlowEval introduces a novel, consensus-based, reference-free dialogue evaluation framework leveraging segment act flows, significantly improving automatic assessment accuracy by modeling dialogue at a finer granularity.

Contribution

It extends dialog act modeling from utterance to segment level and develops a large-scale dataset, enabling more precise and effective dialogue evaluation.

Findings

FlowEval outperforms strong baselines on benchmark datasets.

Segment act flows improve evaluation accuracy.

The framework demonstrates robustness and potential for better dialogue assessment.

Abstract

Despite recent progress in open-domain dialogue evaluation, how to develop automatic metrics remains an open problem. We explore the potential of dialogue evaluation featuring dialog act information, which was hardly explicitly modeled in previous methods. However, defined at the utterance level in general, dialog act is of coarse granularity, as an utterance can contain multiple segments possessing different functions. Hence, we propose segment act, an extension of dialog act from utterance level to segment level, and crowdsource a large-scale dataset for it. To utilize segment act flows, sequences of segment acts, for evaluation, we develop the first consensus-based dialogue evaluation framework, FlowEval. This framework provides a reference-free approach for dialog evaluation by finding pseudo-references. Extensive experiments against strong baselines on three benchmark datasets demonstrate the effectiveness and other desirable characteristics of our FlowEval, pointing out a potential path for better dialogue evaluation.