$\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization

TL;DR

The paper introduces $.5$, a vision-language-action model that leverages heterogeneous data and co-training to achieve broad generalization for robotic manipulation in real-world, open-world environments, enabling tasks like cleaning in new homes.

Contribution

It presents $.5$, a novel model that combines multi-modal data and co-training to enhance generalization in robotic manipulation tasks beyond laboratory settings.

Findings

Enables long-horizon, dexterous manipulation in new environments

Uses diverse data sources for effective knowledge transfer

Achieves real-world kitchen and bedroom cleaning tasks

Abstract

In order for robots to be useful, they must perform practically relevant tasks in the real world, outside of the lab. While vision-language-action (VLA) models have demonstrated impressive results for end-to-end robot control, it remains an open question how far such models can generalize in the wild. We describe $\pi_{0.5}$, a new model based on $\pi_{0}$ that uses co-training on heterogeneous tasks to enable broad generalization. $\pi_{0.5}$\ uses data from multiple robots, high-level semantic prediction, web data, and other sources to enable broadly generalizable real-world robotic manipulation. Our system uses a combination of co-training and hybrid multi-modal examples that combine image observations, language commands, object detections, semantic subtask prediction, and low-level actions. Our experiments show that this kind of knowledge transfer is essential for effective generalization, and we demonstrate for the first time that an end-to-end learning-enabled robotic system can perform long-horizon and dexterous manipulation skills, such as cleaning a kitchen or bedroom, in entirely new homes.