Scaling World Model for Hierarchical Manipulation Policies

TL;DR

This paper introduces a hierarchical vision-language-action framework that leverages a large-scale pre-trained world model to improve robot manipulation in out-of-distribution scenarios, significantly enhancing generalization and performance.

Contribution

The novel hierarchical framework combines a high-level world model with low-level visual-language policies, enabling better generalization across unseen objects and scenarios.

Findings

Performance boost from 14% to 69% in OOD scenarios

Outperforms previous baselines in out-of-distribution tasks

Effective visual goal synthesis improves manipulation success

Abstract

Vision-Language-Action (VLA) models are promising for generalist robot manipulation but remain brittle in out-of-distribution (OOD) settings, especially with limited real-robot data. To resolve the generalization bottleneck, we introduce a hierarchical Vision-Language-Action framework \our{} that leverages the generalization of large-scale pre-trained world model for robust and generalizable VIsual Subgoal TAsk decomposition VISTA. Our hierarchical framework \our{} consists of a world model as the high-level planner and a VLA as the low-level executor. The high-level world model first divides manipulation tasks into subtask sequences with goal images, and the low-level policy follows the textual and visual guidance to generate action sequences. Compared to raw textual goal specification, these synthesized goal images provide visually and physically grounded details for low-level policies, making it feasible to generalize across unseen objects and novel scenarios. We validate both visual goal synthesis and our hierarchical VLA policies in massive out-of-distribution scenarios, and the performance of the same-structured VLA in novel scenarios could boost from 14% to 69% with the guidance generated by the world model. Results demonstrate that our method outperforms previous baselines with a clear margin, particularly in out-of-distribution scenarios. Project page: \href{https://vista-wm.github.io/}{https://vista-wm.github.io}