$\pi^{*}_{0.6}$: a VLA That Learns From Experience

TL;DR

This paper introduces RECAP, a reinforcement learning method that improves vision-language-action models through real-world experience, enabling robots to perform complex tasks like laundry and espresso making with higher efficiency and reliability.

Contribution

The paper presents RECAP, a novel RL approach that integrates heterogeneous data for training and fine-tuning a generalist VLA model called $ ext{pi}^*_{0.6}$ for real-world robotic tasks.

Findings

RECAP significantly increases task throughput.

RECAP reduces task failure rates.

The $ ext{pi}^*_{0.6}$ model successfully performs complex household tasks.

Abstract

We study how vision-language-action (VLA) models can improve through real-world deployments via reinforcement learning (RL). We present a general-purpose method, RL with Experience and Corrections via Advantage-conditioned Policies (RECAP), that provides for RL training of VLAs via advantage conditioning. Our method incorporates heterogeneous data into the self-improvement process, including demonstrations, data from on-policy collection, and expert teleoperated interventions provided during autonomous execution. RECAP starts by pre-training a generalist VLA with offline RL, which we call $\pi^{*}_{0.6}$, that can then be specialized to attain high performance on downstream tasks through on-robot data collection. We show that the $\pi^{*}_{0.6}$ model trained with the full RECAP method can fold laundry in real homes, reliably assemble boxes, and make espresso drinks using a professional espresso machine. On some of the hardest tasks, RECAP more than doubles task throughput and roughly halves the task failure rate.