Where, What, Why: Towards Explainable Driver Attention Prediction

TL;DR

This paper introduces a new explainable framework for driver attention prediction that models where drivers look, what they focus on, and why, supported by a large annotated dataset and a novel LLM-driven architecture.

Contribution

It presents the first large-scale explainable driver attention dataset and a unified LLM-based model for predicting and understanding driver attention mechanisms.

Findings

LLada outperforms existing models in accuracy and robustness.

The dataset W3DA provides detailed semantic and causal annotations.

The approach enhances interpretability of driver attention prediction.

Abstract

Modeling task-driven attention in driving is a fundamental challenge for both autonomous vehicles and cognitive science. Existing methods primarily predict where drivers look by generating spatial heatmaps, but fail to capture the cognitive motivations behind attention allocation in specific contexts, which limits deeper understanding of attention mechanisms. To bridge this gap, we introduce Explainable Driver Attention Prediction, a novel task paradigm that jointly predicts spatial attention regions (where), parses attended semantics (what), and provides cognitive reasoning for attention allocation (why). To support this, we present W3DA, the first large-scale explainable driver attention dataset. It enriches existing benchmarks with detailed semantic and causal annotations across diverse driving scenarios, including normal conditions, safety-critical situations, and traffic accidents. We further propose LLada, a Large Language model-driven framework for driver attention prediction, which unifies pixel modeling, semantic parsing, and cognitive reasoning within an end-to-end architecture. Extensive experiments demonstrate the effectiveness of LLada, exhibiting robust generalization across datasets and driving conditions. This work serves as a key step toward a deeper understanding of driver attention mechanisms, with significant implications for autonomous driving, intelligent driver training, and human-computer interaction.