Multi-View Video Diffusion Policy: A 3D Spatio-Temporal-Aware Video Action Model

TL;DR

This paper introduces MV-VDP, a multi-view video diffusion policy that models 3D spatio-temporal environment dynamics for robotic manipulation, enabling data-efficient, robust, and generalizable actions.

Contribution

The novel MV-VDP approach jointly predicts multi-view heatmap and RGB videos, aligning video pretraining with action finetuning for improved manipulation performance.

Findings

MV-VDP performs complex tasks with only ten demonstrations.

It outperforms existing models on Meta-World and real robots.

The model generalizes well and predicts realistic future videos.

Abstract

Robotic manipulation requires understanding both the 3D spatial structure of the environment and its temporal evolution, yet most existing policies overlook one or both. They typically rely on 2D visual observations and backbones pretrained on static image--text pairs, resulting in high data requirements and limited understanding of environment dynamics. To address this, we introduce MV-VDP, a multi-view video diffusion policy that jointly models the 3D spatio-temporal state of the environment. The core idea is to simultaneously predict multi-view heatmap videos and RGB videos, which 1) align the representation format of video pretraining with action finetuning, and 2) specify not only what actions the robot should take, but also how the environment is expected to evolve in response to those actions. Extensive experiments show that MV-VDP enables data-efficient, robust, generalizable, and interpretable manipulation. With only ten demonstration trajectories and without additional pretraining, MV-VDP successfully performs complex real-world tasks, demonstrates strong robustness across a range of model hyperparameters, generalizes to out-of-distribution settings, and predicts realistic future videos. Experiments on Meta-World and real-world robotic platforms demonstrate that MV-VDP consistently outperforms video-prediction--based, 3D-based, and vision--language--action models, establishing a new state of the art in data-efficient multi-task manipulation.