Open-Domain Dialogue Quality Evaluation: Deriving Nugget-level Scores from Turn-level Scores

TL;DR

This paper introduces a fine-grained dialogue quality evaluation method that decomposes system turns into nuggets for more precise problem localization, leveraging existing turn-level evaluation systems.

Contribution

It proposes a novel approach to derive nugget-level scores from turn-level evaluations, enabling detailed analysis of dialogue quality.

Findings

Nugget-level evaluation provides more precise insights into dialogue issues.

The method effectively leverages existing turn-level evaluation systems.

Case study demonstrates the potential of the approach.

Abstract

Existing dialogue quality evaluation systems can return a score for a given system turn from a particular viewpoint, e.g., engagingness. However, to improve dialogue systems by locating exactly where in a system turn potential problems lie, a more fine-grained evaluation may be necessary. We therefore propose an evaluation approach where a turn is decomposed into nuggets (i.e., expressions associated with a dialogue act), and nugget-level evaluation is enabled by leveraging an existing turn-level evaluation system. We demonstrate the potential effectiveness of our evaluation method through a case study.