Attention-Guided Patch-Wise Sparse Adversarial Attacks on Vision-Language-Action Models

TL;DR

This paper introduces ADVLA, a novel, efficient adversarial attack method that subtly disrupts vision-language-action models by applying sparse, focused perturbations in feature space, achieving high success with minimal perceptibility.

Contribution

ADVLA is the first method to directly perturb features in the textual space of VLA models, avoiding costly training and producing imperceptible, sparse attacks with high success rates.

Findings

Achieves nearly 100% attack success rate under low-amplitude constraints.

Modifies less than 10% of patches, maintaining imperceptibility.

Single-step attack takes approximately 0.06 seconds.

Abstract

In recent years, Vision-Language-Action (VLA) models in embodied intelligence have developed rapidly. However, existing adversarial attack methods require costly end-to-end training and often generate noticeable perturbation patches. To address these limitations, we propose ADVLA, a framework that directly applies adversarial perturbations on features projected from the visual encoder into the textual feature space. ADVLA efficiently disrupts downstream action predictions under low-amplitude constraints, and attention guidance allows the perturbations to be both focused and sparse. We introduce three strategies that enhance sensitivity, enforce sparsity, and concentrate perturbations. Experiments demonstrate that under an $L_{\infty}=4/255$ constraint, ADVLA combined with Top-K masking modifies less than 10% of the patches while achieving an attack success rate of nearly 100%. The perturbations are concentrated on critical regions, remain almost imperceptible in the overall image, and a single-step iteration takes only about 0.06 seconds, significantly outperforming conventional patch-based attacks. In summary, ADVLA effectively weakens downstream action predictions of VLA models under low-amplitude and locally sparse conditions, avoiding the high training costs and conspicuous perturbations of traditional patch attacks, and demonstrates unique effectiveness and practical value for attacking VLA feature spaces.