Detecting Interlocutor Confusion in Situated Human-Avatar Dialogue: A Pilot Study

TL;DR

This pilot study explores detecting user confusion in human-avatar dialogue using deep learning to analyze emotion, head pose, and eye gaze, aiming to improve engagement in conversational systems.

Contribution

It introduces a new confusion definition tailored for dialogue systems and demonstrates the relationship between physiological indicators and confusion states.

Findings

Significant correlation between emotion, head pose, eye gaze, and confusion.

Deep learning models can estimate user confusion from physiological cues.

Pilot results support further research in automated dialogue pragmatics analysis.

Abstract

In order to enhance levels of engagement with conversational systems, our long term research goal seeks to monitor the confusion state of a user and adapt dialogue policies in response to such user confusion states. To this end, in this paper, we present our initial research centred on a user-avatar dialogue scenario that we have developed to study the manifestation of confusion and in the long term its mitigation. We present a new definition of confusion that is particularly tailored to the requirements of intelligent conversational system development for task-oriented dialogue. We also present the details of our Wizard-of-Oz based data collection scenario wherein users interacted with a conversational avatar and were presented with stimuli that were in some cases designed to invoke a confused state in the user. Post study analysis of this data is also presented. Here, three pre-trained deep learning models were deployed to estimate base emotion, head pose and eye gaze. Despite a small pilot study group, our analysis demonstrates a significant relationship between these indicators and confusion states. We understand this as a useful step forward in the automated analysis of the pragmatics of dialogue.