Modeling social interaction dynamics using temporal graph networks

TL;DR

This paper introduces an adapted Temporal Graph Network model that effectively captures social interaction dynamics using multi-modal data, significantly improving prediction accuracy while reducing computational complexity for human-robot interaction tasks.

Contribution

The paper presents a novel, efficient temporal graph network approach for modeling social interactions, incorporating multi-modal data and outperforming baseline models in prediction tasks.

Findings

F1-score improved by 37.0% over baseline

Next speaker prediction improved by 29.0%

Reduced message passing from 768 to 14 elements

Abstract

Integrating intelligent systems, such as robots, into dynamic group settings poses challenges due to the mutual influence of human behaviors and internal states. A robust representation of social interaction dynamics is essential for effective human-robot collaboration. Existing approaches often narrow their focus to facial expressions or speech, overlooking the broader context. We propose employing an adapted Temporal Graph Networks to comprehensively represent social interaction dynamics while enabling its practical implementation. Our method incorporates temporal multi-modal behavioral data including gaze interaction, voice activity and environmental context. This representation of social interaction dynamics is trained as a link prediction problem using annotated gaze interaction data. The F1-score outperformed the baseline model by 37.0%. This improvement is consistent for a secondary task of next speaker prediction which achieves an improvement of 29.0%. Our contributions are two-fold, including a model to representing social interaction dynamics which can be used for many downstream human-robot interaction tasks like human state inference and next speaker prediction. More importantly, this is achieved using a more concise yet efficient message passing method, significantly reducing it from 768 to 14 elements, while outperforming the baseline model.