Understanding Adversarial Transferability in Vision-Language Models for Autonomous Driving: A Cross-Architecture Analysis

TL;DR

This study systematically investigates how physical adversarial patches can transfer across different vision-language models used in autonomous driving, revealing high transferability and potential security risks.

Contribution

It provides the first comprehensive cross-architecture analysis of adversarial transferability in VLMs for autonomous driving, using real-world patches in diverse scenarios.

Findings

High transfer rates of 73-91% across models.

Physical patches can manipulate decisions over 64.7-79.4% of critical windows.

Transferability persists even without model-specific patch optimization.

Abstract

Vision-language models (VLMs) are increasingly used in autonomous driving because they combine visual perception with language-based reasoning, supporting more interpretable decision-making, yet their robustness to physical adversarial attacks, especially whether such attacks transfer across different VLM architectures, is not well understood and poses a practical risk when attackers do not know which model a vehicle uses. We address this gap with a systematic cross-architecture study of adversarial transferability in VLM-based driving, evaluating three representative architectures (Dolphins, OmniDrive, and LeapVAD) using physically realizable patches placed on roadside infrastructure in both crosswalk and highway scenarios. Our transfer-matrix evaluation shows high cross-architecture effectiveness, with transfer rates of 73-91% (mean TR = 0.815 for crosswalk and 0.833 for highway) and sustained frame-level manipulation over 64.7-79.4% of the critical decision window even when patches are not optimized for the target model.