User Experience Estimation in Human-Robot Interaction Via Multi-Instance Learning of Multimodal Social Signals

TL;DR

This paper introduces a Transformer-based multi-instance learning approach that uses multimodal social signals to estimate user experience in human-robot interaction, capturing temporal dynamics for more accurate assessment.

Contribution

It presents a novel multi-instance learning framework leveraging facial expressions and voice for holistic UX estimation in HRI, outperforming human evaluators.

Findings

Our model surpasses third-party evaluators in accuracy.

Captures both short- and long-term interaction patterns.

Utilizes multimodal social signals for comprehensive UX assessment.

Abstract

In recent years, the demand for social robots has grown, requiring them to adapt their behaviors based on users' states. Accurately assessing user experience (UX) in human-robot interaction (HRI) is crucial for achieving this adaptability. UX is a multi-faceted measure encompassing aspects such as sentiment and engagement, yet existing methods often focus on these individually. This study proposes a UX estimation method for HRI by leveraging multimodal social signals. We construct a UX dataset and develop a Transformer-based model that utilizes facial expressions and voice for estimation. Unlike conventional models that rely on momentary observations, our approach captures both short- and long-term interaction patterns using a multi-instance learning framework. This enables the model to capture temporal dynamics in UX, providing a more holistic representation. Experimental results demonstrate that our method outperforms third-party human evaluators in UX estimation.