VLATest: Testing and Evaluating Vision-Language-Action Models for Robotic Manipulation

TL;DR

This paper introduces VLATest, a fuzzing framework for testing vision-language-action models in robotic manipulation, revealing their current lack of robustness across diverse scenarios and conditions.

Contribution

We developed VLATest to generate diverse testing scenes and conducted an empirical study on seven VLA models, exposing their robustness limitations.

Findings

VLA models lack robustness in diverse scenarios

Performance drops with confounding objects and lighting changes

Unseen objects and instruction mutations significantly affect accuracy

Abstract

The rapid advancement of generative AI and multi-modal foundation models has shown significant potential in advancing robotic manipulation. Vision-language-action (VLA) models, in particular, have emerged as a promising approach for visuomotor control by leveraging large-scale vision-language data and robot demonstrations. However, current VLA models are typically evaluated using a limited set of hand-crafted scenes, leaving their general performance and robustness in diverse scenarios largely unexplored. To address this gap, we present VLATest, a fuzzing framework designed to generate robotic manipulation scenes for testing VLA models. Based on VLATest, we conducted an empirical study to assess the performance of seven representative VLA models. Our study results revealed that current VLA models lack the robustness necessary for practical deployment. Additionally, we investigated the impact of various factors, including the number of confounding objects, lighting conditions, camera poses, unseen objects, and task instruction mutations, on the VLA model's performance. Our findings highlight the limitations of existing VLA models, emphasizing the need for further research to develop reliable and trustworthy VLA applications.