Unmasking the Illusion of Embodied Reasoning in Vision-Language-Action Models

TL;DR

This paper introduces BeTTER, a diagnostic benchmark revealing that current vision-language-action models fail in dynamic scenarios due to architectural bottlenecks, despite high success rates on standard benchmarks.

Contribution

The paper presents BeTTER, a novel diagnostic benchmark that uncovers fundamental architectural limitations in current VLA models, emphasizing the need for improved reasoning capabilities.

Findings

State-of-the-art VLAs fail catastrophically in dynamic scenarios.

Architectural bottlenecks degrade semantic representations.

Static evaluation masks underlying model deficiencies.

Abstract

Recent Vision-Language-Action (VLA) models report impressive success rates on standard robotic benchmarks, fueling optimism about general-purpose physical intelligence. However, recent evidence suggests a systematic misalignment between standard benchmark success and true embodied reasoning, raising the question of whether these high scores reflect genuine cognitive capability. To address this gap, we introduce BeTTER, a diagnostic Benchmark for Testing True Embodied Reasoning in robotic policies. BeTTER applies targeted causal interventions (e.g., spatial layout shifts, temporal extrapolation) while enforcing kinematic isolation to explicitly decouple high-level reasoning failures from low-level execution limits. Through systematic evaluation, we reveal that state-of-the-art VLAs catastrophically fail in dynamic scenarios, exhibiting severe lexical-kinematic shortcuts, behavioral inertia, and semantic feature collapse. Crucially, our mechanistic analysis traces these symptoms to fundamental architectural bottlenecks - such as capacity compression and myopic downsampling - which systematically degrade the model's foundational semantic representation. We demonstrate that highly static evaluation protocols effectively mask this degradation by allowing optimization to overfit to sensorimotor priors. Supported by real-world robotic validation, our findings confirm that this representational breakdown is not a simulation artifact, highlighting the critical need for future VLA paradigms to resolve the structural tension between high-frequency control and high-level reasoning.