A Novel Framework for Multi-Person Temporal Gaze Following and Social Gaze Prediction

TL;DR

This paper introduces a unified, transformer-based framework for joint multi-person gaze following and social gaze prediction, leveraging a new dataset to improve generalization and performance in understanding social interactions.

Contribution

The paper presents a novel joint model and a new dataset, VSGaze, enabling simultaneous prediction of gaze targets and social gaze labels for multiple individuals.

Findings

Achieves state-of-the-art results on multi-person gaze following

Successfully predicts social gaze labels in complex scenes

Demonstrates the effectiveness of a unified temporal transformer approach

Abstract

Gaze following and social gaze prediction are fundamental tasks providing insights into human communication behaviors, intent, and social interactions. Most previous approaches addressed these tasks separately, either by designing highly specialized social gaze models that do not generalize to other social gaze tasks or by considering social gaze inference as an ad-hoc post-processing of the gaze following task. Furthermore, the vast majority of gaze following approaches have proposed static models that can handle only one person at a time, therefore failing to take advantage of social interactions and temporal dynamics. In this paper, we address these limitations and introduce a novel framework to jointly predict the gaze target and social gaze label for all people in the scene. The framework comprises of: (i) a temporal, transformer-based architecture that, in addition to image tokens, handles person-specific tokens capturing the gaze information related to each individual; (ii) a new dataset, VSGaze, that unifies annotation types across multiple gaze following and social gaze datasets. We show that our model trained on VSGaze can address all tasks jointly, and achieves state-of-the-art results for multi-person gaze following and social gaze prediction.