ReinboT: Amplifying Robot Visual-Language Manipulation with Reinforcement Learning

TL;DR

ReinboT is a novel vision-language-action model for robots that leverages reinforcement learning to improve decision-making, robustness, and generalization in manipulation tasks, especially with mixed-quality training data.

Contribution

The paper introduces ReinboT, integrating RL with VLA models to enhance data understanding and decision robustness in robotic manipulation.

Findings

Achieves state-of-the-art results on CALVIN dataset

Exhibits superior few-shot learning capabilities

Shows improved out-of-distribution generalization

Abstract

Vision-Language-Action (VLA) models have shown great potential in general robotic decision-making tasks via imitation learning. However, the variable quality of training data often constrains the performance of these models. On the other hand, offline Reinforcement Learning (RL) excels at learning robust policy models from mixed-quality data. In this paper, we introduce Reinforced robot GPT (ReinboT), a novel end-to-end VLA model that integrates the RL principle of maximizing cumulative reward. ReinboT achieves a deeper understanding of the data quality distribution by predicting dense returns that capture the nuances of manipulation tasks. The dense return prediction capability enables the robot to generate more robust decision-making actions, oriented towards maximizing future benefits. Extensive experiments show that ReinboT achieves state-of-the-art performance on the CALVIN mixed-quality dataset and exhibits superior few-shot learning and out-of-distribution generalization capabilities in real-world tasks.