RL-VLA$^3$: A Flexible and Asynchronous Reinforcement Learning Framework for VLA Training

TL;DR

RL-VLA$^3$ is a novel asynchronous reinforcement learning framework designed for VLA training, significantly improving throughput and scalability while handling variable latencies in simulation environments.

Contribution

It introduces the first fully asynchronous RL framework specifically tailored for VLA training, addressing system-level challenges and enhancing performance.

Findings

Achieves up to 85.2% throughput improvement over synchronous methods.

Maintains sample efficiency comparable to traditional approaches.

Scales effectively from 8 to 256 GPUs.

Abstract

Reinforcement learning (RL) has emerged as a critical paradigm for post-training Vision-Language-Action (VLA) models, enabling embodied agents to adapt and improve through environmental interaction. However, existing RL frameworks for VLAs inherit synchronous design principles from traditional LLM training, treating entire rollouts as indivisible units and alternating strictly between data collection and policy optimization. This fundamentally mismatches the unique characteristics of VLA training, as physical simulators introduce highly variable, resource-intensive latencies. To address this, we introduce RL-VLA$^3$, a fully asynchronous distributed RL framework that enables fine-grained asynchronous interaction between simulation, inference, and training components through dynamic batching schedulers and flexible environment sharding strategies. Extensive experiments across diverse simulation backends, VLA architectures, and RL algorithms demonstrate that RL-VLA$^3$ achieves throughput improvements of up to 85.2\% over synchronous baselines while maintaining identical sample efficiency, with scalability validated from 8 to 256 GPUs. To our knowledge, RL-VLA$^3$ is the first fully asynchronous RL training framework tailored specifically for the system-level challenges of VLA training.