Joint Engagement Classification using Video Augmentation Techniques for Multi-person Human-robot Interaction

TL;DR

This paper introduces a hybrid deep learning framework utilizing video augmentation techniques to improve joint engagement recognition in multi-person human-robot interactions, specifically focusing on parent-child dyads in a home setting.

Contribution

It presents a novel combination of video augmentation methods with deep learning models for joint engagement classification in multi-person interactions, enhancing interpretability and performance.

Findings

Augmentation techniques improved classification accuracy.

Models successfully recognized joint engagement in real interaction scenarios.

Introduced a behavior-based metric for model interpretability.

Abstract

Affect understanding capability is essential for social robots to autonomously interact with a group of users in an intuitive and reciprocal way. However, the challenge of multi-person affect understanding comes from not only the accurate perception of each user's affective state (e.g., engagement) but also the recognition of the affect interplay between the members (e.g., joint engagement) that presents as complex, but subtle, nonverbal exchanges between them. Here we present a novel hybrid framework for identifying a parent-child dyad's joint engagement by combining a deep learning framework with various video augmentation techniques. Using a dataset of parent-child dyads reading storybooks together with a social robot at home, we first train RGB frame- and skeleton-based joint engagement recognition models with four video augmentation techniques (General Aug, DeepFake, CutOut, and Mixed) applied datasets to improve joint engagement classification performance. Second, we demonstrate experimental results on the use of trained models in the robot-parent-child interaction context. Third, we introduce a behavior-based metric for evaluating the learned representation of the models to investigate the model interpretability when recognizing joint engagement. This work serves as the first step toward fully unlocking the potential of end-to-end video understanding models pre-trained on large public datasets and augmented with data augmentation and visualization techniques for affect recognition in the multi-person human-robot interaction in the wild.