AcceRL: A Distributed Asynchronous Reinforcement Learning and World Model Framework for Vision-Language-Action Models

TL;DR

AcceRL introduces a fully asynchronous, distributed RL framework with an integrated trainable world model, significantly improving efficiency, scalability, and sample utilization for vision-language-action models.

Contribution

It is the first to incorporate a plug-and-play trainable world model into a distributed asynchronous RL pipeline, enhancing virtual experience generation and training efficiency.

Findings

Achieves state-of-the-art performance on LIBERO benchmark.

Exhibits super-linear throughput scaling and efficient hardware use.

Provides unprecedented sample efficiency and training stability in complex tasks.

Abstract

Reinforcement learning (RL) for large-scale Vision-Language-Action (VLA) models faces significant challenges in computational efficiency and data acquisition. We propose AcceRL, a fully asynchronous and decoupled RL framework designed to eliminate synchronization barriers by physically isolating training, inference, and rollouts. Crucially, AcceRL is the first to integrate a plug-and-play, trainable world model into a distributed asynchronous RL pipeline to generate virtual experiences. Experiments on the LIBERO~\cite{liu2023libero} benchmark demonstrate that AcceRL achieves state-of-the-art (SOTA) performance. Systematically, it exhibits super-linear scaling in throughput and highly efficient hardware utilization. Algorithmically, the world-model-augmented variant delivers unprecedented sample efficiency and robust training stability in complex control tasks. Code is publicly available at https://github.com/distanceLu/AcceRL.