VideoWorld 2: Learning Transferable Knowledge from Real-world Videos

TL;DR

VideoWorld 2 introduces a novel method for learning transferable task knowledge directly from raw real-world videos, significantly improving performance in robotic manipulation tasks and enabling coherent long-horizon reasoning.

Contribution

It proposes a dynamic-enhanced Latent Dynamics Model that decouples action dynamics from visual appearance using a pretrained diffusion model, advancing video-based transfer learning.

Findings

Achieves up to 70% improvement in task success rate

Produces coherent long execution videos

Enhances robotic manipulation performance

Abstract

Learning transferable knowledge from unlabeled video data and applying it in new environments is a fundamental capability of intelligent agents. This work presents VideoWorld 2, which extends VideoWorld and offers the first investigation into learning transferable knowledge directly from raw real-world videos. At its core, VideoWorld 2 introduces a dynamic-enhanced Latent Dynamics Model (dLDM) that decouples action dynamics from visual appearance: a pretrained video diffusion model handles visual appearance modeling, enabling the dLDM to learn latent codes that focus on compact and meaningful task-related dynamics. These latent codes are then modeled autoregressively to learn task policies and support long-horizon reasoning. We evaluate VideoWorld 2 on challenging real-world handcraft making tasks, where prior video generation and latent-dynamics models struggle to operate reliably. Remarkably, VideoWorld 2 achieves up to 70% improvement in task success rate and produces coherent long execution videos. In robotics, we show that VideoWorld 2 can acquire effective manipulation knowledge from the Open-X dataset, which substantially improves task performance on CALVIN. This study reveals the potential of learning transferable world knowledge directly from raw videos, with all code, data, and models to be open-sourced for further research.