Robot Learning from a Physical World Model

TL;DR

PhysWorld introduces a framework that combines video generation and physical world modeling to enable robots to learn manipulation tasks from visual demonstrations without real robot data, achieving zero-shot generalization.

Contribution

This work presents PhysWorld, a novel approach coupling video synthesis with physical reconstruction to improve robot manipulation accuracy from visual cues.

Findings

Significantly improves manipulation accuracy over previous methods

Enables zero-shot generalization in robotic manipulation tasks

Eliminates the need for real robot data collection

Abstract

We introduce PhysWorld, a framework that enables robot learning from video generation through physical world modeling. Recent video generation models can synthesize photorealistic visual demonstrations from language commands and images, offering a powerful yet underexplored source of training signals for robotics. However, directly retargeting pixel motions from generated videos to robots neglects physics, often resulting in inaccurate manipulations. PhysWorld addresses this limitation by coupling video generation with physical world reconstruction. Given a single image and a task command, our method generates task-conditioned videos and reconstructs the underlying physical world from the videos, and the generated video motions are grounded into physically accurate actions through object-centric residual reinforcement learning with the physical world model. This synergy transforms implicit visual guidance into physically executable robotic trajectories, eliminating the need for real robot data collection and enabling zero-shot generalizable robotic manipulation. Experiments on diverse real-world tasks demonstrate that PhysWorld substantially improves manipulation accuracy compared to previous approaches. Visit \href{https://pointscoder.github.io/PhysWorld_Web/}{the project webpage} for details.