DRAW2ACT: Turning Depth-Encoded Trajectories into Robotic Demonstration Videos

TL;DR

DRAW2ACT is a depth-aware, trajectory-conditioned video generation framework that produces controllable, consistent robotic demonstration videos by integrating multiple modalities and depth information, improving manipulation success.

Contribution

It introduces a novel depth-aware, multimodal diffusion model for generating controllable robotic videos and a policy conditioned on these videos for improved manipulation.

Findings

Achieves higher visual fidelity and consistency.

Yields higher manipulation success rates.

Outperforms existing baselines.

Abstract

Video diffusion models provide powerful real-world simulators for embodied AI but remain limited in controllability for robotic manipulation. Recent works on trajectory-conditioned video generation address this gap but often rely on 2D trajectories or single modality conditioning, which restricts their ability to produce controllable and consistent robotic demonstrations. We present DRAW2ACT, a depth-aware trajectory-conditioned video generation framework that extracts multiple orthogonal representations from the input trajectory, capturing depth, semantics, shape and motion, and injects them into the diffusion model. Moreover, we propose to jointly generate spatially aligned RGB and depth videos, leveraging cross-modality attention mechanisms and depth supervision to enhance the spatio-temporal consistency. Finally, we introduce a multimodal policy model conditioned on the generated RGB and depth sequences to regress the robot's joint angles. Experiments on Bridge V2, Berkeley Autolab, and simulation benchmarks show that DRAW2ACT achieves superior visual fidelity and consistency while yielding higher manipulation success rates compared to existing baselines.