From Scene to Object: Text-Guided Dual-Gaze Prediction

TL;DR

This paper introduces a novel framework for fine-grained, object-level driver attention prediction using a new dataset and a dual-branch model, significantly improving spatial accuracy and interpretability.

Contribution

It constructs G-W3DA, an object-level driver attention dataset, and proposes DualGaze-VLM, a model that achieves precise, intent-driven attention prediction surpassing existing methods.

Findings

DualGaze-VLM outperforms SOTA models with up to 17.8% improvement in Similarity (SIM).

Attention heatmaps from DualGaze-VLM are perceived as authentic by 88.22% of human evaluators.

The dataset and model enable more accurate and interpretable driver attention prediction.

Abstract

Interpretable driver attention prediction is crucial for human-like autonomous driving. However, existing datasets provide only scene-level global gaze rather than fine-grained object-level annotations, inherently failing to support text-grounded cognitive modeling. Consequently, while Vision-Language Models (VLMs) hold great potential for semantic reasoning, this critical data limitations leads to severe text-vision decoupling and visual-bias hallucinations. To break this bottleneck and achieve precise object-level attention prediction, this paper proposes a novel dual-branch gaze prediction framework, establishing a complete paradigm from data construction to model architecture. First, we construct G-W3DA, a object-level driver attention dataset. By integrating a multimodal large language model with the Segment Anything Model 3 (SAM3), we decouple macroscopic heatmaps into object-level masks under rigorous cross-validation, fundamentally eliminating annotation hallucinations. Building upon this high-quality data foundation, we propose the DualGaze-VLM architecture. This architecture extracts the hidden states of semantic queries and dynamically modulates visual features via a Condition-Aware SE-Gate, achieving intent-driven precise spatial anchoring. Extensive experiments on the W3DA benchmark demonstrate that DualGaze-VLM consistently surpasses existing state-of-the-art (SOTA) models in spatial alignment metrics, notably achieving up to a 17.8% improvement in Similarity (SIM) under safety-critical scenarios. Furthermore, a visual Turing test reveals that the attention heatmaps generated by DualGaze-VLM are perceived as authentic by 88.22% of human evaluators, proving its capability to generate rational cognitive priors.