Is the User Enjoying the Conversation? A Case Study on the Impact on the Reward Function

TL;DR

This paper introduces a deep neural network approach using semantic representations to estimate user satisfaction in dialogue systems, significantly improving reward inference and task success rates.

Contribution

It proposes a hierarchical neural network model that outperforms existing quality estimators and enhances reward function inference in dialogue policy learning.

Findings

Hierarchical network outperforms state-of-the-art estimators.

Semantic representation-based models improve satisfaction estimation.

Enhanced reward inference leads to higher task success rates.

Abstract

The impact of user satisfaction in policy learning task-oriented dialogue systems has long been a subject of research interest. Most current models for estimating the user satisfaction either (i) treat out-of-context short-texts, such as product reviews, or (ii) rely on turn features instead of on distributed semantic representations. In this work we adopt deep neural networks that use distributed semantic representation learning for estimating the user satisfaction in conversations. We evaluate the impact of modelling context length in these networks. Moreover, we show that the proposed hierarchical network outperforms state-of-the-art quality estimators. Furthermore, we show that applying these networks to infer the reward function in a Partial Observable Markov Decision Process (POMDP) yields to a great improvement in the task success rate.