Understanding User Satisfaction with Task-oriented Dialogue Systems

TL;DR

This paper investigates how various user experience aspects influence satisfaction ratings in task-oriented dialogue systems by annotating dialogues at multiple levels and analyzing their impact on user satisfaction.

Contribution

It introduces a detailed annotation scheme for dialogue aspects and demonstrates their significance in understanding user satisfaction beyond overall ratings.

Findings

Relevance and interestingness significantly influence satisfaction for different users.

Annotations reveal diverse satisfaction factors across dialogues and annotators.

Fine-grained analysis uncovers nuances in user satisfaction not captured by overall ratings.

Abstract

$ $Dialogue systems are evaluated depending on their type and purpose. Two categories are often distinguished: (1) task-oriented dialogue systems (TDS), which are typically evaluated on utility, i.e., their ability to complete a specified task, and (2) open domain chatbots, which are evaluated on the user experience, i.e., based on their ability to engage a person. What is the influence of user experience on the user satisfaction rating of TDS as opposed to, or in addition to, utility? We collect data by providing an additional annotation layer for dialogues sampled from the ReDial dataset, a widely used conversational recommendation dataset. Unlike prior work, we annotate the sampled dialogues at both the turn and dialogue level on six dialogue aspects: relevance, interestingness, understanding, task completion, efficiency, and interest arousal. The annotations allow us to study how different dialogue aspects influence user satisfaction. We introduce a comprehensive set of user experience aspects derived from the annotators' open comments that can influence users' overall impression. We find that the concept of satisfaction varies across annotators and dialogues, and show that a relevant turn is significant for some annotators, while for others, an interesting turn is all they need. Our analysis indicates that the proposed user experience aspects provide a fine-grained analysis of user satisfaction that is not captured by a monolithic overall human rating.