VLAW: Iterative Co-Improvement of Vision-Language-Action Policy and World Model

TL;DR

This paper introduces an iterative method that enhances vision-language-action models by using a learned world model to generate synthetic data, leading to significant performance improvements on real robot tasks.

Contribution

It proposes a simple iterative algorithm that leverages real-world data to improve a learned world model, which then generates synthetic data to boost VLA model performance.

Findings

39.2% success rate improvement over base policy

11.6% improvement from synthetic rollouts

Effective enhancement of VLA models on real robot tasks

Abstract

The goal of this paper is to improve the performance and reliability of vision-language-action (VLA) models through iterative online interaction. Since collecting policy rollouts in the real world is expensive, we investigate whether a learned simulator-specifically, an action-conditioned video generation model-can be used to generate additional rollout data. Unfortunately, existing world models lack the physical fidelity necessary for policy improvement: they are predominantly trained on demonstration datasets that lack coverage of many different physical interactions (particularly failure cases) and struggle to accurately model small yet critical physical details in contact-rich object manipulation. We propose a simple iterative improvement algorithm that uses real-world roll-out data to improve the fidelity of the world model, which can then, in turn, be used to generate supplemental synthetic data for improving the VLA model. In our experiments on a real robot, we use this approach to improve the performance of a state-of-the-art VLA model on multiple downstream tasks. We achieve a 39.2% absolute success rate improvement over the base policy and 11.6% improvement from training with the generated synthetic rollouts. Videos can be found at this anonymous website: https://sites.google.com/view/vla-w