Gaze-VLM:Bridging Gaze and VLMs through Attention Regularization for Egocentric Understanding

TL;DR

Gaze-VLM introduces a gaze-regularized attention mechanism during training to improve egocentric understanding tasks in vision-language models, enhancing future event prediction and activity comprehension.

Contribution

The paper presents a novel gaze-regularized training framework that aligns model attention with human gaze, improving VLM performance in egocentric tasks without using gaze at inference.

Findings

Up to 11% improvement in future event prediction accuracy

Approximately 7% enhancement in current activity understanding

Gaze-guided training boosts model robustness and accuracy

Abstract

Eye gaze offers valuable cues about attention, short-term intent, and future actions, making it a powerful signal for modeling egocentric behavior. In this work, we propose a gaze-regularized framework that enhances VLMs for two key egocentric understanding tasks: fine-grained future event prediction and current activity understanding. Unlike prior approaches that rely solely on visual inputs or use gaze as an auxiliary input signal , our method uses gaze only during training. We introduce a gaze-regularized attention mechanism that aligns model focus with human visual gaze. This design is flexible and modular, allowing it to generalize across multiple VLM architectures that utilize attention. Experimental results show that our approach improves semantic prediction scores by up to 11 for future event prediction and around 7 for current activity understanding, compared to the corresponding baseline models trained without gaze regularization. These results highlight the value of gaze-guided training in improving the accuracy and robustness of egocentric VLMs. Overall, this work establishes a foundation for using human gaze to enhance the predictive capabilities of VLMs in real-world scenarios like assistive robots and human-machine collaboration. Code and additional information is available at: https://github.com/anupampani/Gaze-VLM