FSDAM: Few-Shot Driving Attention Modeling via Vision-Language Coupling

TL;DR

FSDAM is a few-shot learning framework that predicts driver attention and generates structured explanations by decomposing attention into reasoning components, using minimal annotated data to enhance interpretability and generalization in autonomous driving.

Contribution

The paper introduces FSDAM, a novel dual-pathway architecture for joint attention prediction and explanation generation with minimal supervision, addressing data scarcity and task interference.

Findings

Achieves competitive gaze prediction performance with only 90 annotations.

Generates coherent, context-aware explanations for attention shifts.

Demonstrates strong zero-shot generalization across multiple benchmarks.

Abstract

Understanding not only where drivers look but also why their attention shifts is essential for interpretable human-AI collaboration in autonomous driving. Driver attention is not purely perceptual but semantically structured. Thus, attention shifts can be learned through minimal semantic supervision rather than dense large-scale annotation. We present \textbf{FSDAM} (\textbf{F}ew-\textbf{S}hot \textbf{D}river \textbf{A}ttention \textbf{M}odeling), a framework that achieves joint spatial attention prediction and structured explanation generation using 90 annotated examples. Our key insight is to decompose attention into an explicit reasoning representation, including scene context, current focus, anticipated next focus, and causal explanation, and to learn next-focus anticipation through minimal-pair supervision. To address task conflict and large sample requirements of existing models, and to mitigate task interference under limited data, we introduce a novel dual-pathway architecture in which separate modules handle spatial prediction and caption generation. In addition, we use a training-only vision-language alignment mechanism that injects semantic priors into spatial learning without increasing inference complexity, mitigating task interference under few-shot training. Despite extreme data scarcity, FSDAM achieves competitive performance in gaze prediction, and generates coherent, context-aware structural reasoning for improved interpretability. The model further demonstrates strong zero-shot generalization across multiple driving benchmarks.