Discover, Learn, and Reinforce: Scaling Vision-Language-Action Pretraining with Diverse RL-Generated Trajectories

TL;DR

This paper introduces DLR, a pattern discovery framework that enhances the diversity of RL-generated trajectories for vision-language-action pretraining, leading to improved downstream performance and scalable data generation.

Contribution

DLR enables the generation of multiple distinct, high-success behavioral patterns in RL, significantly increasing trajectory diversity for VLA pretraining compared to standard RL methods.

Findings

DLR produces more diverse trajectories than standard RL.

Pretraining on DLR data improves downstream task performance.

DLR exhibits positive data-scaling behavior.

Abstract

Scaling vision-language-action (VLA) model pre-training requires large volumes of diverse, high-quality manipulation trajectories. Most current data is obtained via human teleoperation, which is expensive and difficult to scale. Reinforcement learning (RL) methods learn useful skills through autonomous exploration, making them a viable approach for generating data. However, standard RL training collapses to a narrow execution pattern, limiting its utility for large-scale pre-training. We propose Discover, Lea rn and Reinforce (DLR), an information-theoretic pattern discovery framework that generates multiple distinct, high-success behavioral patterns for VLA pretraining. Empirically, DLR generates a markedly more diverse trajectory corpus on LIBERO. Specifically, it learns multiple distinct, high-success strategies for the same task where standard RL discovers only one, and hence it covers substantially broader regions of the state-action space. When adapted to unseen downstream task suites, VLA models pretrained on our diverse RL data surpass counterparts trained on equal-sized standard RL datasets. Moreover, DLR exhibits positive data-scaling behavior that single-pattern RL lacks. These results position multi-pattern RL as a practical, scalable data engine for embodied foundation models.