LIBERO-X: Robustness Litmus for Vision-Language-Action Models

TL;DR

LIBERO-X introduces a hierarchical benchmark with diverse training data to better evaluate the robustness and generalization of vision-language-action models under real-world complexities.

Contribution

It proposes a comprehensive benchmarking framework combining progressive evaluation protocols and diverse datasets to improve assessment of VLA models.

Findings

Models show significant performance drops under complex perturbations.

Hierarchical evaluation reveals specific weaknesses in scene understanding.

Diverse training data helps bridge the gap between training and real-world scenarios.

Abstract

Reliable benchmarking is critical for advancing Vision-Language-Action (VLA) models, as it reveals their generalization, robustness, and alignment of perception with language-driven manipulation tasks. However, existing benchmarks often provide limited or misleading assessments due to insufficient evaluation protocols that inadequately capture real-world distribution shifts. This work systematically rethinks VLA benchmarking from both evaluation and data perspectives, introducing LIBERO-X, a benchmark featuring: 1) A hierarchical evaluation protocol with progressive difficulty levels targeting three core capabilities: spatial generalization, object recognition, and task instruction understanding. This design enables fine-grained analysis of performance degradation under increasing environmental and task complexity; 2) A high-diversity training dataset collected via human teleoperation, where each scene supports multiple fine-grained manipulation objectives to bridge the train-evaluation distribution gap. Experiments with representative VLA models reveal significant performance drops under cumulative perturbations, exposing persistent limitations in scene comprehension and instruction grounding. By integrating hierarchical evaluation with diverse training data, LIBERO-X offers a more reliable foundation for assessing and advancing VLA development.