VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Models

TL;DR

VLA-Arena is a comprehensive, open-source benchmark framework designed to evaluate vision-language-action models across diverse, fine-grained tasks, revealing key limitations and fostering future research in generalist robot policies.

Contribution

We introduce VLA-Arena, a structured, multi-dimensional benchmark with a novel task design framework and extensive evaluation tools for assessing VLAs' capabilities and robustness.

Findings

VLAs tend to memorize rather than generalize.

Robustness is asymmetric across perturbations.

Current models struggle with safety and long-horizon tasks.

Abstract

While Vision-Language-Action models (VLAs) are rapidly advancing towards generalist robot policies, it remains difficult to quantitatively understand their limits and failure modes. To address this, we introduce a comprehensive benchmark called VLA-Arena. We propose a novel structured task design framework to quantify difficulty across three orthogonal axes: (1) Task Structure, (2) Language Command, and (3) Visual Observation. This allows us to systematically design tasks with fine-grained difficulty levels, enabling a precise measurement of model capability frontiers. For Task Structure, VLA-Arena's 170 tasks are grouped into four dimensions: Safety, Distractor, Extrapolation, and Long Horizon. Each task is designed with three difficulty levels (L0-L2), with fine-tuning performed exclusively on L0 to assess general capability. Orthogonal to this, language (W0-W4) and visual (V0-V4) perturbations can be applied to any task to enable a decoupled analysis of robustness. Our extensive evaluation of state-of-the-art VLAs reveals several critical limitations, including a strong tendency toward memorization over generalization, asymmetric robustness, a lack of consideration for safety constraints, and an inability to compose learned skills for long-horizon tasks. To foster research addressing these challenges and ensure reproducibility, we provide the complete VLA-Arena framework, including an end-to-end toolchain from task definition to automated evaluation and the VLA-Arena-S/M/L datasets for fine-tuning. Our benchmark, data, models, and leaderboard are available at https://vla-arena.github.io.