CSGaze: Context-aware Social Gaze Prediction

TL;DR

CSGaze is a novel multimodal model that uses contextual cues, facial, and scene information to predict social gaze patterns during conversations, improving accuracy and interpretability over existing methods.

Contribution

Introduces CSGaze, a context-aware multimodal approach with a fine-grained attention mechanism for enhanced social gaze prediction and explainability.

Findings

Performs competitively with state-of-the-art methods on multiple datasets.

Highlights the importance of contextual cues in social gaze prediction.

Demonstrates robustness and generalizability across diverse scenarios.

Abstract

A person's gaze offers valuable insights into their focus of attention, level of social engagement, and confidence. In this work, we investigate how contextual cues combined with visual scene and facial information can be effectively utilized to predict and interpret social gaze patterns during conversational interactions. We introduce CSGaze, a context aware multimodal approach that leverages facial, scene information as complementary inputs to enhance social gaze pattern prediction from multi-person images. The model also incorporates a fine-grained attention mechanism centered on the principal speaker, which helps in better modeling social gaze dynamics. Experimental results show that CSGaze performs competitively with state-of-the-art methods on GP-Static, UCO-LAEO and AVA-LAEO. Our findings highlight the role of contextual cues in improving social gaze prediction. Additionally, we provide initial explainability through generated attention scores, offering insights into the model's decision-making process. We also demonstrate our model's generalizability by testing our model on open set datasets that demonstrating its robustness across diverse scenarios.