Robust Understanding of Human-Robot Social Interactions through Multimodal Distillation

TL;DR

This paper introduces a robust, efficient multimodal knowledge distillation framework for social understanding in human-robot interactions, achieving high accuracy even with noisy or incomplete data.

Contribution

It presents a novel knowledge distillation approach that models social cues from multiple modalities and produces a lightweight, robust student model for real-time social scene analysis.

Findings

Student model outperforms baselines by 14.75% accuracy.

Maintains high performance with up to 51% corrupted input.

Student model is less than 1% the size of the teacher and 11.9% in latency.

Abstract

There is a growing need for social robots and intelligent agents that can effectively interact with and support users. For the interactions to be seamless, the agents need to analyse social scenes and behavioural cues from their (robot's) perspective. Works that model human-agent interactions in social situations are few; and even those existing ones are computationally too intensive to be deployed in real time or perform poorly in real-world scenarios when only limited information is available. We propose a knowledge distillation framework that models social interactions through various multimodal cues, and yet is robust against incomplete and noisy information during inference. We train a teacher model with multimodal input (body, face and hand gestures, gaze, raw images) that transfers knowledge to a student model which relies solely on body pose. Extensive experiments on two publicly available human-robot interaction datasets demonstrate that our student model achieves an average accuracy gain of 14.75% over competitive baselines on multiple downstream social understanding tasks, even with up to 51% of its input being corrupted. The student model is also highly efficient - less than 1% in size of the teacher model in terms of parameters and its latency is 11.9% of the teacher model. Our code and related data are available at github.com/biantongfei/SocialEgoMobile.