VL4Gaze: Unleashing Vision-Language Models for Gaze Following

TL;DR

VL4Gaze introduces a large-scale benchmark to evaluate and enhance vision-language models' ability to understand gaze, revealing current limitations and the benefits of targeted training for gaze-related tasks.

Contribution

This paper presents VL4Gaze, the first comprehensive benchmark for gaze understanding in vision-language models, and demonstrates the importance of task-specific supervision for improving gaze interpretation.

Findings

Large-scale VLMs struggle with gaze semantics without supervision.

Training on VL4Gaze significantly improves gaze understanding.

Targeted multi-task training enhances VLMs' gaze-related capabilities.

Abstract

Human gaze provides essential cues for interpreting attention, intention, and social interaction in visual scenes, yet gaze understanding remains largely unexplored in current vision-language models (VLMs). While recent VLMs achieve strong scene-level reasoning across a range of visual tasks, there exists no benchmark that systematically evaluates or trains them for gaze interpretation, leaving open the question of whether gaze understanding can emerge from general-purpose vision-language pre-training. To address this gap, we introduce VL4Gaze, the first large-scale benchmark designed to investigate, evaluate, and unlock the potential of VLMs for gaze understanding. VL4Gaze contains 489K automatically generated question-answer pairs across 124K images and formulates gaze understanding as a unified VQA problem through four complementary tasks: (1) gaze object description, (2) gaze direction description, (3) gaze point location, and (4) ambiguous question recognition. We comprehensively evaluate both commercial and open-source VLMs under in-context learning and fine-tuning settings. The results show that even large-scale VLMs struggle to reliably infer gaze semantics and spatial localization without task-specific supervision. In contrast, training on VL4Gaze brings substantial and consistent improvements across all tasks, highlighting the importance of targeted multi-task supervision for developing gaze understanding capabilities in VLMs. We will release the dataset and code to support further research and development in this direction.