User Satisfaction Estimation with Sequential Dialogue Act Modeling in Goal-oriented Conversational Systems

TL;DR

This paper introduces USDA, a framework that models sequential dialogue acts to improve user satisfaction estimation in goal-oriented systems, using a hierarchical transformer and joint learning of dialogue act recognition and satisfaction prediction.

Contribution

It proposes a novel sequential dialogue act modeling framework with task-adaptive pre-training and both supervised and unsupervised variants, enhancing user satisfaction estimation accuracy.

Findings

USDA outperforms existing methods on four benchmark datasets.

Sequential dialogue act modeling significantly improves satisfaction prediction.

The approach validates the importance of dialogue act sequences in user satisfaction estimation.

Abstract

User Satisfaction Estimation (USE) is an important yet challenging task in goal-oriented conversational systems. Whether the user is satisfied with the system largely depends on the fulfillment of the user's needs, which can be implicitly reflected by users' dialogue acts. However, existing studies often neglect the sequential transitions of dialogue act or rely heavily on annotated dialogue act labels when utilizing dialogue acts to facilitate USE. In this paper, we propose a novel framework, namely USDA, to incorporate the sequential dynamics of dialogue acts for predicting user satisfaction, by jointly learning User Satisfaction Estimation and Dialogue Act Recognition tasks. In specific, we first employ a Hierarchical Transformer to encode the whole dialogue context, with two task-adaptive pre-training strategies to be a second-phase in-domain pre-training for enhancing the dialogue modeling ability. In terms of the availability of dialogue act labels, we further develop two variants of USDA to capture the dialogue act information in either supervised or unsupervised manners. Finally, USDA leverages the sequential transitions of both content and act features in the dialogue to predict the user satisfaction. Experimental results on four benchmark goal-oriented dialogue datasets across different applications show that the proposed method substantially and consistently outperforms existing methods on USE, and validate the important role of dialogue act sequences in USE.