VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning

TL;DR

VLA-RL introduces a reinforcement learning framework that enhances vision-language-action models for robotic manipulation, improving performance and robustness in out-of-distribution scenarios through online adaptation.

Contribution

The paper presents a novel RL-based approach for fine-tuning pretrained VLA models, including a trajectory-level formulation and reward modeling, to improve robotic manipulation in diverse tasks.

Findings

VLA-RL outperforms finetuned baselines by 4.5% on LIBERO tasks.

It matches the performance of advanced commercial models.

Test-time optimization benefits from increased inference scaling.

Abstract

Recent high-capacity vision-language-action (VLA) models have demonstrated impressive performance on a range of robotic manipulation tasks by imitating human demonstrations. However, exploiting offline data with limited visited states will cause execution failure in out-of-distribution scenarios. Intuitively, an exploration-based method that improves on online collected data at test time could address this limitation. We present VLA-RL, an algorithmic and systematic framework that leverages online reinforcement learning (RL) to improve pretrained auto-regressive VLAs in downstream tasks. Within a unified perspective, we first introduce a trajectory-level RL formulation for auto-regressive VLA training, which models general robotic manipulation trajectory as multi-modal multi-turn conversation. To address the challenge of sparse rewards, we fine-tune a pretrained vision-language model as a robotic process reward model, which is trained on pseudo reward labels annotated on automatically extracted task segments. To scale up, we identify several implementation findings that improve the stability and efficiency including curriculum selection strategy, GPU-balanced vectorized environments, batch decoding, and critic warmup. VLA-RL enables OpenVLA-7B to surpass the strongest finetuned baseline by 4.5% on 40 challenging robotic manipulation tasks in LIBERO, and even matches the performance of advanced commercial models such as $\pi_0$-FAST. Notably, we observe that VLA-RL benefits from increased test-time optimization, indicating an early spark of inference scaling laws in robotics.