Interact with me: Joint Egocentric Forecasting of Intent to Interact, Attitude and Social Actions

TL;DR

This paper introduces SocialEgoNet, a real-time, graph-based model that jointly forecasts human intent, attitude, and actions from egocentric video, enhancing human-agent interaction understanding.

Contribution

The paper presents a novel hierarchical multitask learning framework, SocialEgoNet, that predicts multiple social cues from short video clips, with improved accuracy and speed over existing methods.

Findings

Achieved 83.15% average accuracy across tasks

Demonstrated real-time inference capability

Outperformed several baseline models in experiments

Abstract

For efficient human-agent interaction, an agent should proactively recognize their target user and prepare for upcoming interactions. We formulate this challenging problem as the novel task of jointly forecasting a person's intent to interact with the agent, their attitude towards the agent and the action they will perform, from the agent's (egocentric) perspective. So we propose \emph{SocialEgoNet} - a graph-based spatiotemporal framework that exploits task dependencies through a hierarchical multitask learning approach. SocialEgoNet uses whole-body skeletons (keypoints from face, hands and body) extracted from only 1 second of video input for high inference speed. For evaluation, we augment an existing egocentric human-agent interaction dataset with new class labels and bounding box annotations. Extensive experiments on this augmented dataset, named JPL-Social, demonstrate \emph{real-time} inference and superior performance (average accuracy across all tasks: 83.15\%) of our model outperforming several competitive baselines. The additional annotations and code will be available upon acceptance.