RL Token: Bootstrapping Online RL with Vision-Language-Action Models

TL;DR

This paper presents RLT, a method for efficient online reinforcement learning fine-tuning of pretrained vision-language-action models using an RL token, significantly improving robot task performance within hours.

Contribution

The introduction of the RL token interface enables rapid, sample-efficient online RL fine-tuning of large VLAs for real-world robotic tasks.

Findings

RLT improves task speed by up to 3x on the hardest parts.

Success rates increase significantly within minutes to hours.

RLT can outperform human teleoperation in some tasks.

Abstract

Vision-language-action (VLA) models can learn to perform diverse manipulation skills "out of the box," but achieving the precision and speed that real-world tasks demand requires further fine-tuning -- for example, via reinforcement learning (RL). We introduce a lightweight method that enables sample-efficient online RL fine-tuning of pretrained VLAs using just a few hours of real-world practice. We (1) adapt the VLA to expose an "RL token," a compact readout representation that preserves task-relevant pretrained knowledge while serving as an efficient interface for online RL, and (2) train a small actor-critic head on this RL token to refine the actions, while anchoring the learned policy to the VLA. Online RL with the RL token (RLT) makes it possible to fine-tune even large VLAs with RL quickly and efficiently. Across four real-robot tasks (screw installation, zip tie fastening, charger insertion, and Ethernet insertion), RLT improves the speed on the hardest part of the task by up to 3x and raises success rates significantly within minutes to a few hours of practice. It can even surpass the speed of human teleoperation on some of the tasks.