Video2Act: A Dual-System Video Diffusion Policy with Robotic Spatio-Motional Modeling

TL;DR

Video2Act introduces a dual-system framework that leverages video diffusion models for more coherent and physically consistent robotic action learning, significantly improving success rates in simulation and real-world tasks.

Contribution

It presents a novel dual-system approach integrating spatial and motion-aware representations from VDMs with a diffusion transformer for enhanced robotic policy learning.

Findings

Surpasses previous state-of-the-art methods by 7.7% in simulation

Achieves 21.7% higher success rate in real-world tasks

Demonstrates strong generalization capabilities across tasks

Abstract

Robust perception and dynamics modeling are fundamental to real-world robotic policy learning. Recent methods employ video diffusion models (VDMs) to enhance robotic policies, improving their understanding and modeling of the physical world. However, existing approaches overlook the coherent and physically consistent motion representations inherently encoded across frames in VDMs. To this end, we propose Video2Act, a framework that efficiently guides robotic action learning by explicitly integrating spatial and motion-aware representations. Building on the inherent representations of VDMs, we extract foreground boundaries and inter-frame motion variations while filtering out background noise and task-irrelevant biases. These refined representations are then used as additional conditioning inputs to a diffusion transformer (DiT) action head, enabling it to reason about what to manipulate and how to move. To mitigate inference inefficiency, we propose an asynchronous dual-system design, where the VDM functions as the slow System 2 and the DiT head as the fast System 1, working collaboratively to generate adaptive actions. By providing motion-aware conditions to System 1, Video2Act maintains stable manipulation even with low-frequency updates from the VDM. For evaluation, Video2Act surpasses previous state-of-the-art VLA methods by 7.7% in simulation and 21.7% in real-world tasks in terms of average success rate, further exhibiting strong generalization capabilities.