Robust Skills, Brittle Grounding: Diagnosing Restricted Generalization in Vision-Language Action Policies via Multi-Object Picking

TL;DR

This paper investigates whether vision-language action policies truly understand object grounding or rely on superficial correlations, revealing that primitive manipulation skills are more robust than instruction following in challenging scenarios.

Contribution

The study introduces controlled stress tests for VLA policies, demonstrating the decoupling of manipulation skills from instruction grounding and proposing improved benchmarking methods.

Findings

Manipulation primitives are more reliable than instruction-conditioned success in complex settings.

VLA policies rely on object-location correlations that do not transfer beyond training.

Augmenting benchmarks with task ladders improves diagnosis of instruction-grounded generalization.

Abstract

Vision-language action (VLA) policies often report strong manipulation benchmark performance with relatively few demonstrations, but it remains unclear whether this reflects robust language-to-object grounding or reliance on object--location correlations that do not transfer beyond the training distribution. We present a controlled multi-object picking study that progressively increases object placement variability up to full workspace randomization and evaluates held-out object--location pairings that break familiar associations without increasing spatial difficulty. Across these stress tests and data scaling, we find that for representative VLA policies, including SmolVLA and $\pi_{0.5}$, execution of the manipulation primitive remains substantially more reliable than instruction-conditioned task success in harder regimes, suggesting that manipulation skill acquisition is decoupled from instruction following. We recommend augmenting manipulation benchmarks with task ladders and decomposed metrics that separately measure primitive execution and instruction-conditioned success to better diagnose instruction-grounded generalization.