Towards Affect-Adaptive Human-Robot Interaction: A Protocol for Multimodal Dataset Collection on Social Anxiety

TL;DR

This paper proposes a protocol for collecting a multimodal dataset involving audio, video, and physiological signals during human-robot interactions to study social anxiety, aiming to improve affect detection in social robotics.

Contribution

It introduces a comprehensive protocol for creating a multimodal social anxiety dataset in human-robot interaction settings, addressing a gap in existing data resources.

Findings

Dataset will include synchronized multimodal recordings from 70+ participants.

Participants will engage in 10-minute social interactions with a social robot.

The dataset aims to enhance affect detection methods for social anxiety.

Abstract

Social anxiety is a prevalent condition that affects interpersonal interactions and social functioning. Recent advances in artificial intelligence and social robotics offer new opportunities to examine social anxiety in the human-robot interaction context. Accurate detection of affective states and behaviours associated with social anxiety requires multimodal datasets, where each signal modality provides complementary insights into its manifestations. However, such datasets remain scarce, limiting progress in both research and applications. To address this, this paper presents a protocol for multimodal dataset collection designed to reflect social anxiety in a human-robot interaction context. The dataset will consist of synchronised audio, video, and physiological recordings acquired from at least 70 participants, grouped according to their level of social anxiety, as they engage in approximately 10-minute interactive Wizard-of-Oz role-play scenarios with the Furhat social robot under controlled experimental conditions. In addition to multimodal data, the dataset will be enriched with contextual data providing deeper insight into individual variability in social anxiety responses. This work can contribute to research on affect-adaptive human-robot interaction by providing support for robust multimodal detection of social anxiety.