Sound-Based Recognition of Touch Gestures and Emotions for Enhanced Human-Robot Interaction

TL;DR

This paper introduces a lightweight audio-based model that recognizes touch gestures and emotional states during human-robot interactions, addressing privacy concerns and hardware limitations of humanoid robots.

Contribution

It presents a novel, small-scale audio-only recognition model for tactile gestures and emotions, suitable for privacy-preserving and resource-constrained robotic applications.

Findings

Effective recognition of arousal and valence in emotions

Accurate tactile gesture classification from sound data

Comparable performance to larger pretrained audio models

Abstract

Emotion recognition and touch gesture decoding are crucial for advancing human-robot interaction (HRI), especially in social environments where emotional cues and tactile perception play important roles. However, many humanoid robots, such as Pepper, Nao, and Furhat, lack full-body tactile skin, limiting their ability to engage in touch-based emotional and gesture interactions. In addition, vision-based emotion recognition methods usually face strict GDPR compliance challenges due to the need to collect personal facial data. To address these limitations and avoid privacy issues, this paper studies the potential of using the sounds produced by touching during HRI to recognise tactile gestures and classify emotions along the arousal and valence dimensions. Using a dataset of tactile gestures and emotional interactions from 28 participants with the humanoid robot Pepper, we design an audio-only lightweight touch gesture and emotion recognition model with only 0.24M parameters, 0.94MB model size, and 0.7G FLOPs. Experimental results show that the proposed sound-based touch gesture and emotion recognition model effectively recognises the arousal and valence states of different emotions, as well as various tactile gestures, when the input audio length varies. The proposed model is low-latency and achieves similar results as well-known pretrained audio neural networks (PANNs), but with much smaller FLOPs, parameters, and model size.