DreamPlan: Efficient Reinforcement Fine-Tuning of Vision-Language Planners via Video World Models

TL;DR

DreamPlan introduces a reinforcement fine-tuning framework for vision-language models using video world models, enabling efficient physical grounding and improved manipulation success without extensive real-world data.

Contribution

It proposes a novel method to fine-tune VLMs via virtual rollouts in a learned video world model, reducing reliance on costly real-world interactions.

Findings

Significantly improves manipulation success rates.

Efficiently injects physical knowledge into VLMs.

Reduces need for large-scale real-world data.

Abstract

Robotic manipulation requires sophisticated commonsense reasoning, a capability naturally possessed by large-scale Vision-Language Models (VLMs). While VLMs show promise as zero-shot planners, their lack of grounded physical understanding often leads to compounding errors and low success rates when deployed in complex real-world environments, particularly for challenging tasks like deformable object manipulation. Although Reinforcement Learning (RL) can adapt these planners to specific task dynamics, directly fine-tuning VLMs via real-world interaction is prohibitively expensive, unsafe, and sample-inefficient. To overcome this bottleneck, we introduce DreamPlan, a novel framework for the reinforcement fine-tuning of VLM planners via video world models. Instead of relying on costly physical rollouts, DreamPlan first leverages the zero-shot VLM to collect exploratory interaction data. We demonstrate that this sub-optimal data is sufficient to train an action-conditioned video generation model, which implicitly captures complex real-world physics. Subsequently, the VLM planner is fine-tuned entirely within the "imagination" of this video world model using Odds Ratio Policy Optimization (ORPO). By utilizing these virtual rollouts, physical and task-specific knowledge is efficiently injected into the VLM. Our results indicate that DreamPlan bridges the gap between semantic reasoning and physical grounding, significantly improving manipulation success rates without the need for large-scale real-world data collection. Our project page is https://psi-lab.ai/DreamPlan/.