A Multimodal Neural Network for Recognizing Subjective Self-Disclosure Towards Social Robots

TL;DR

This paper presents a multimodal neural network designed to recognize subjective self-disclosure in social interactions, especially with robots, using a novel loss function and a large video corpus, achieving high accuracy.

Contribution

The study introduces a new multimodal attention network and a scale preserving cross entropy loss for improved self-disclosure recognition in social robots.

Findings

Achieved an F1 score of 0.83 with the proposed model.

Developed a large self-disclosure video corpus for training.

Introduced a novel loss function that outperforms existing methods.

Abstract

Subjective self-disclosure is an important feature of human social interaction. While much has been done in the social and behavioural literature to characterise the features and consequences of subjective self-disclosure, little work has been done thus far to develop computational systems that are able to accurately model it. Even less work has been done that attempts to model specifically how human interactants self-disclose with robotic partners. It is becoming more pressing as we require social robots to work in conjunction with and establish relationships with humans in various social settings. In this paper, our aim is to develop a custom multimodal attention network based on models from the emotion recognition literature, training this model on a large self-collected self-disclosure video corpus, and constructing a new loss function, the scale preserving cross entropy loss, that improves upon both classification and regression versions of this problem. Our results show that the best performing model, trained with our novel loss function, achieves an F1 score of 0.83, an improvement of 0.48 from the best baseline model. This result makes significant headway in the aim of allowing social robots to pick up on an interaction partner's self-disclosures, an ability that will be essential in social robots with social cognition.