Comparative Analysis of Patch Attack on VLM-Based Autonomous Driving Architectures

TL;DR

This paper systematically evaluates the robustness of three vision-language model architectures for autonomous driving against physical patch attacks, revealing significant vulnerabilities and architectural differences in their susceptibility.

Contribution

It introduces a comparative framework for adversarial evaluation of VLM-based autonomous driving models, highlighting their vulnerabilities and architectural weaknesses.

Findings

All architectures show severe vulnerabilities to patch attacks.

Multi-frame failures are common across models.

Object detection performance degrades critically under attack.

Abstract

Vision-language models are emerging for autonomous driving, yet their robustness to physical adversarial attacks remains unexplored. This paper presents a systematic framework for comparative adversarial evaluation across three VLM architectures: Dolphins, OmniDrive (Omni-L), and LeapVAD. Using black-box optimization with semantic homogenization for fair comparison, we evaluate physically realizable patch attacks in CARLA simulation. Results reveal severe vulnerabilities across all architectures, sustained multi-frame failures, and critical object detection degradation. Our analysis exposes distinct architectural vulnerability patterns, demonstrating that current VLM designs inadequately address adversarial threats in safety-critical autonomous driving applications.