On the Use of Linguistic Features for the Evaluation of Generative Dialogue Systems

TL;DR

This paper explores using linguistic features as an interpretable, reference-free metric for evaluating dialogue systems, aiming to better correlate with human judgment and generalize across domains.

Contribution

It introduces a linguistic feature-based evaluation method that does not rely on gold standards or human annotations, demonstrating its effectiveness and generalization capabilities.

Findings

Features align with known properties of dialogue models

Method shows promising zero-shot domain generalization

Features correlate well with human judgment

Abstract

Automatically evaluating text-based, non-task-oriented dialogue systems (i.e., `chatbots') remains an open problem. Previous approaches have suffered challenges ranging from poor correlation with human judgment to poor generalization and have often required a gold standard reference for comparison or human-annotated data. Extending existing evaluation methods, we propose that a metric based on linguistic features may be able to maintain good correlation with human judgment and be interpretable, without requiring a gold-standard reference or human-annotated data. To support this proposition, we measure and analyze various linguistic features on dialogues produced by multiple dialogue models. We find that the features' behaviour is consistent with the known properties of the models tested, and is similar across domains. We also demonstrate that this approach exhibits promising properties such as zero-shot generalization to new domains on the related task of evaluating response relevance.