EnDex: Evaluation of Dialogue Engagingness at Scale

TL;DR

EnDex is a novel human-reaction based model for evaluating dialogue engagingness, trained on a large Reddit dataset, addressing data scarcity and capturing high-level quality aligned with user experience.

Contribution

It introduces the first human-reaction based evaluation metric for dialogue engagingness, trained on a large dataset with a novel distant-supervision framework, and provides theoretical and empirical validation.

Findings

High correlation with engagingness datasets

Effective in capturing user-perceived quality

Outperforms synthetic-based metrics

Abstract

We propose EnDex, the first human-reaction based model to evaluate dialogue engagingness. EnDex is trained on 80k Reddit-based Engagement Dataset (RED) curated using a novel distant-supervision framework. Engagingness is a key measure that captures high-level quality of AI dialogue systems and closely reflects actual user experience. However, data shortage, plus the abstract and extensive definition of engagingness makes it challenging to develop an automatic metric. Our work departs from mainstream approaches that use synthetic negative examples to train binary classifiers, and instead, proposes a solution using distant-supervision from human-reaction feedback. To support the soundness of our EnDex metric, we offer a theoretical foundation for engagement, an extensive ablation study, and empirical evidence of high correlation on five engagingness related datasets. We will release code, off-the-shelf EnDex model, and a large-scale dataset upon paper publication to facilitate future research.