DEUS: A Data-driven Approach to Estimate User Satisfaction in Multi-turn Dialogues

TL;DR

This paper introduces DEUS, a data-driven, context-sensitive method for estimating user satisfaction at each turn in multi-turn dialogues, considering user preferences and interaction costs, validated through experiments.

Contribution

The paper presents a novel approach to evaluate turn-level user satisfaction in multi-turn dialogues using a budget consumption model, addressing scalability and context-awareness.

Findings

Effective estimation of turn-level satisfaction demonstrated

Validated on simulated and real dialogue data

Outperforms existing evaluation methods

Abstract

Digital assistants are experiencing rapid growth due to their ability to assist users with day-to-day tasks where most dialogues are happening multi-turn. However, evaluating multi-turn dialogues remains challenging, especially at scale. We suggest a context-sensitive method to estimate the turn-level satisfaction for dialogue considering various types of user preferences. The costs of interactions between users and dialogue systems are formulated using a budget consumption concept. We assume users have an initial interaction budget for a dialogue formed based on the task complexity and that each turn has a cost. When the task is completed, or the budget has been exhausted, users quit the dialogue. We demonstrate our method's effectiveness by extensive experimentation with a simulated dialogue platform and real multi-turn dialogues.