Classification of User Satisfaction in HRI with Social Signals in the Wild

TL;DR

This paper presents a method for automatically classifying user satisfaction in human-robot interactions by analyzing social signals like facial expressions and body pose, using time series classification on in-the-wild data.

Contribution

It introduces a novel approach to automatically assess user satisfaction from social signals in real-world interactions, reducing reliance on manual annotations.

Findings

Effective identification of low satisfaction interactions

Use of social signals for real-time satisfaction classification

Potential for improved autonomous evaluation of SIAs

Abstract

Socially interactive agents (SIAs) are being used in various scenarios and are nearing productive deployment. Evaluating user satisfaction with SIAs' performance is a key factor in designing the interaction between the user and SIA. Currently, subjective user satisfaction is primarily assessed manually through questionnaires or indirectly via system metrics. This study examines the automatic classification of user satisfaction through analysis of social signals, aiming to enhance both manual and autonomous evaluation methods for SIAs. During a field trial at the Deutsches Museum Bonn, a Furhat Robotics head was employed as a service and information hub, collecting an "in-the-wild" dataset. This dataset comprises 46 single-user interactions, including questionnaire responses and video data. Our method focuses on automatically classifying user satisfaction based on time series classification. We use time series of social signal metrics derived from the body pose, time series of facial expressions, and physical distance. This study compares three feature engineering approaches on different machine learning models. The results confirm the method's effectiveness in reliably identifying interactions with low user satisfaction without the need for manually annotated datasets. This approach offers significant potential for enhancing SIA performance and user experience through automated feedback mechanisms.