Motion as Emotion: Detecting Affect and Cognitive Load from Free-Hand Gestures in VR

TL;DR

This paper introduces Motion as Emotion, a method that detects user affect and cognitive load in VR through analysis of free-hand gestures, achieving accurate predictions without extra sensors.

Contribution

It presents a novel approach to infer affect and cognitive load from hand motion in VR using gesture features and standard classifiers, without additional hardware.

Findings

Gesture features significantly differ with affect and load levels

Support vector classifiers accurately predict affect and load levels

Method works with standard VR setups without extra sensors

Abstract

Affect and cognitive load influence many user behaviors. In this paper, we propose Motion as Emotion, a novel method that utilizes fine differences in hand motion to recognise affect and cognitive load in virtual reality (VR). We conducted a study with 22 participants who used common free-hand gesture interactions to carry out tasks of varying difficulty in VR environments. We find that the affect and cognitive load induced by tasks are associated with significant differences in gesture features such as speed, distance and hand tension. Standard support vector classification (SVC) models could accurately predict two levels (low, high) of valence, arousal and cognitive load from these features. Our results demonstrate the potential of Motion as Emotion as an accurate and reliable method of inferring user affect and cognitive load from free-hand gestures, without needing any additional wearable sensors or modifications to a standard VR headset.