Follow My Eyes: Backdoor Attacks on VLM-based Scanpath Prediction

TL;DR

This paper explores backdoor attacks on vision-language model-based scanpath prediction, demonstrating effective, diverse attacks that evade detection and survive deployment, posing security risks for gaze-driven systems.

Contribution

It introduces novel variable-output backdoor attacks on VLM-based scanpath prediction that are resilient against defenses and practical for edge devices.

Findings

Naive fixed-path attacks are detectable due to clustering.

Proposed input-aware and duration attacks produce diverse, plausible scanpaths.

Attacks remain effective after quantization and on various smartphones.

Abstract

Scanpath prediction models forecast the sequence and timing of human fixations during visual search, driving foveated rendering and attention-based interaction in mobile systems where their integrity is a first-class security concern. We present the first study of backdoor attacks against VLM-based scanpath prediction, evaluated on GazeFormer and COCO-Search18. We show that naive fixed-path attacks, while effective, create detectable clustering in the continuous output space. To overcome this, we design two variable-output attacks: an input-aware spatial attack that redirects predicted fixations toward an attacker-chosen target object, and a scanpath duration attack that inflates fixation durations to delay visual search completion. Both attacks condition their output on the input scene, producing diverse and plausible scanpaths that evade cluster-based detection. We evaluate across three trigger modalities (visual, textual, and multimodal), multiple poisoning ratios, and five post-training defenses, finding that no defense simultaneously suppresses the attacks and preserves clean performance across all configurations. We further demonstrate that backdoor behavior survives quantization and deployment on both flagship and legacy commodity smartphones, confirming practical threat viability for edge-deployed gaze-driven systems.