Benchmarking Affordance Generalization with BusyBox

TL;DR

This paper introduces BusyBox, a physical benchmark to evaluate the ability of vision-language-action models to generalize affordance manipulation across diverse object variations, highlighting significant challenges even for advanced models.

Contribution

The paper presents BusyBox, a new physical benchmark with diverse object variations for systematic evaluation of affordance generalization in VLAs, along with open resources for the research community.

Findings

Generalization across BusyBox variants is highly challenging for strong VLAs.

BusyBox is easy to build and replicate in most robotics labs.

The dataset includes language-annotated demonstrations with a mobile robot.

Abstract

Vision-Language-Action (VLA) models have been attracting the attention of researchers and practitioners thanks to their promise of generalization. Although single-task policies still offer competitive performance, VLAs are increasingly able to handle commands and environments unseen in their training set. While generalization in vision and language space is undoubtedly important for robust versatile behaviors, a key meta-skill VLAs need to possess is affordance generalization -- the ability to manipulate new objects with familiar physical features. In this work, we present BusyBox, a physical benchmark for systematic semi-automatic evaluation of VLAs' affordance generalization. BusyBox consists of 6 modules with switches, sliders, wires, buttons, a display, and a dial. The modules can be swapped and rotated to create a multitude of BusyBox variations with different visual appearances but the same set of affordances. We empirically demonstrate that generalization across BusyBox variants is highly challenging even for strong open-weights VLAs such as $\pi_{0.5}$ and GR00T-N1.6. To encourage the research community to evaluate their own VLAs on BusyBox and to propose new affordance generalization experiments, we have designed BusyBox to be easy to build in most robotics labs. We release the full set of CAD files for 3D-printing its parts as well as a bill of materials for (optionally) assembling its electronics. We also publish a dataset of language-annotated demonstrations that we collected using the common bimanual Mobile Aloha robot on the canonical BusyBox configuration. All of the released materials are available at https://microsoft.github.io/BusyBox.