Say, Dream, and Act: Learning Video World Models for Instruction-Driven Robot Manipulation

TL;DR

This paper introduces a novel framework for robot manipulation that predicts future environment states using adapted video generation models, enabling more accurate and coherent planning and action execution.

Contribution

It presents a fast, predictive video-conditioned action framework combining adapted video models, adversarial distillation, and correction mechanisms for improved manipulation.

Findings

Enhanced temporal coherence and spatial accuracy in predictions

Significant improvements in task success rates

Better embodiment consistency and spatial referencing

Abstract

Robotic manipulation requires anticipating how the environment evolves in response to actions, yet most existing systems lack this predictive capability, often resulting in errors and inefficiency. While Vision-Language Models (VLMs) provide high-level guidance, they cannot explicitly forecast future states, and existing world models either predict only short horizons or produce spatially inconsistent frames. To address these challenges, we propose a framework for fast and predictive video-conditioned action. Our approach first selects and adapts a robust video generation model to ensure reliable future predictions, then applies adversarial distillation for fast, few-step video generation, and finally trains an action model that leverages both generated videos and real observations to correct spatial errors. Extensive experiments show that our method produces temporally coherent, spatially accurate video predictions that directly support precise manipulation, achieving significant improvements in embodiment consistency, spatial referring ability, and task completion over existing baselines. Codes & Models will be released.