Phantom Menace: Exploring and Enhancing the Robustness of VLA Models Against Physical Sensor Attacks

TL;DR

This paper systematically investigates the vulnerabilities of Vision-Language-Action models to physical sensor attacks, introduces a simulation framework for such attacks, and proposes an adversarial training defense to improve robustness.

Contribution

It is the first comprehensive study of physical sensor attacks on VLA models, including a novel simulation framework and a defense strategy to enhance robustness.

Findings

Vulnerabilities vary with task types and model designs.

Sensor attacks significantly degrade VLA performance.

Adversarial training improves robustness against physical perturbations.

Abstract

Vision-Language-Action (VLA) models revolutionize robotic systems by enabling end-to-end perception-to-action pipelines that integrate multiple sensory modalities, such as visual signals processed by cameras and auditory signals captured by microphones. This multi-modality integration allows VLA models to interpret complex, real-world environments using diverse sensor data streams. Given the fact that VLA-based systems heavily rely on the sensory input, the security of VLA models against physical-world sensor attacks remains critically underexplored. To address this gap, we present the first systematic study of physical sensor attacks against VLAs, quantifying the influence of sensor attacks and investigating the defenses for VLA models. We introduce a novel "Real-Sim-Real" framework that automatically simulates physics-based sensor attack vectors, including six attacks targeting cameras and two targeting microphones, and validates them on real robotic systems. Through large-scale evaluations across various VLA architectures and tasks under varying attack parameters, we demonstrate significant vulnerabilities, with susceptibility patterns that reveal critical dependencies on task types and model designs. We further develop an adversarial-training-based defense that enhances VLA robustness against out-of-distribution physical perturbations caused by sensor attacks while preserving model performance. Our findings expose an urgent need for standardized robustness benchmarks and mitigation strategies to secure VLA deployments in safety-critical environments.