Measuring Conversational Fluidity in Automated Dialogue Agents

TL;DR

This paper introduces an automated evaluation method for conversational fluidity in dialogue systems, combining NLP tools and human ratings to create a classifier that outperforms existing metrics.

Contribution

It presents a novel automated approach that integrates NLP tools and human judgments to better measure fluidity in dialogue agents.

Findings

The new metric improves upon existing fluidity measurement methods.

The classifier correlates well with human judgments.

Experimental results demonstrate enhanced accuracy in fluidity assessment.

Abstract

We present an automated evaluation method to measure fluidity in conversational dialogue systems. The method combines various state of the art Natural Language tools into a classifier, and human ratings on these dialogues to train an automated judgment model. Our experiments show that the results are an improvement on existing metrics for measuring fluidity.